The remarkable thing is that the Viking contacts apparently didn't cause mass deaths. Perhaps the Vikings were too few, and landed in relatively lightly populated areas.
Not only what you said, but all humans around the world were at one time or the other victims of disease. The population of Europe declined by about 30 to 60% according to historians during the Black Death of the 1300s. The fleas which caused the disease came in from the Orient. It took Europe 150 years to recover. Should we blame the Chinese for slaughtering millions of Europeans? Obviously that idea would be laughed off.