A good for-instance is the American Indians. They had no immunity to pox diseases and died in droves, even if they were far from the white men who carried the diseases to the New World.
I don't have a problem with the disease scenario per se, I just don't see how a very sparse human population could spread it all around North America, from tropics to ice floes, from east to west, in such a short time. Also left to the imagination is some idea of how all of these very different animals were somehow vulnerable to a virus that had probably never seen their RNA before.
That is not the problem that you think. The Europeans were very few at first and they did spread the sicknesses throughout the Americas. The black plague went through Europe in just 2-3 years. All humans and all species have contact with their neighbors. This is not as unlikely as it sounds.