A shrinking and shifting of the temperate zones would lead to mass starvation as well. Humanity has very little to fear from global warming. Global cooling would be catastrophic on all levels.
The polar ice would creep toward the equator causing mass migration and overcrowding everywhere. The ice mass increase leads to more sun reflection and ever more cooler atmosphere which severely impairs food growth. Famine of biblical proportions then becomes worldwide. The poorer nations could lose large percentages of their populations as food becomes rationed by price and ability to pay.
You are absolutely correct. Even with a mini ice age, we would be praying for global warming.
Global warming would involve more rainfall globally, which would improve agriculture. It would also make vast stretches of Canada and Siberia more temperate, again improving agricultural output.
Global cooling would kill farming in the northern MidWest.