Continuous long term increases in prosperity depend on economic progress, which depends on increased production, which depends on technological progress, increases in productivity of labor, increased capital accumulation, expansion of division of labor, which increase efficiency. These depend on rationality, freedom, and respect for other natural rights, and the other institutions of capitalism such as profits, price system, freedom of inequality, competition, and freedom of trade.
I agree with everything you say here. But you left out an important factor -- the explanation of which may be Adam Smith's most important contribution to our understanding of how an economy prospers. That factor is specialization.
For example, if every family had to make their own shoes, we'd all be the poorer, due to the lack of specialization. We'd spend maybe a quarter or more of our time making shoes.
Therefore, we all are led by the invisible hand of the market to specialize at what we do best, and we buy our shoes from somebody else.
Ditto for every product and service. And ditto for international trade. All trade, whether domestic or international, is about specialization -- a factor that in the long run will make everybody and every country more prosperous than they would otherwise be.