Wouldn’t that generation of children be pretty inbred?”
They would have been inbred in each location where they survived in small numbers. However, as they multiplied, spread out and encountered other humans, crossbreeding would have occurred in various combinations.
Regarding Ice Ages. We have been having one roughly every 100,000 year for the past million years. There are good charts out there on this. In fact, we are about due for one now or within the next 1,000 years. Since we are only beginning to study the role of interstellar objects and megavolcanoes on our climate, it is hard to say what will cause the next crisis that kills most of us off. Still harder is to predict when.
One of those big 'babies' could blow tonight and then tomorrow...well, you know. (I'll be looking for that Norway seed vault, lol)