Japan, iirc, has an inefficient and highly-subsidized agricultural sector. They have land; it's just not going to the highest value use, as determined by the free market.
However, putting that aside, even if the hypothetical 50% of the population (practically all elderly) is still more crowded than I'd find comfortable, that's not going to defend them against an invader.
It is VERY SMART of them. Otherwise their survival would be at risk in case of some conflict or disruption of imports.