The only reason a population stabilization or decline is viewed as a “disaster” is because it means the Ponzi Scheme of the Welfare State that most “Western” countries (including Japan) have been saddled with collapses unless there is a continuous supply of young workers to prop it up. In the past, countries have gone through periods of population stabilization or moderate decline and it has not been an issue.
It’s a little more complicated than that.
If you build infrastucture for a large population ... but then the population shrinks in size you still have to maintain that infrastructure, but with a much smaller tax base.