The Vikings left Sweden. Why shouldn't they? Sweden is a cold and harsh place. Some migrated to northern France, where they were known as the Normen (North-men). From there, in 1066, they invaded England and became the aristocracy there.
So who was left in Sweden? The slaves the Vikings brought back from elsewhere, and the ones too timid to go adventuring.
Same thing that happened to the Romans. The original Romans marched all around the world. Many settled elsewhere. Meanwhile, millions of slaves were imported into Rome from elsewhere.
A cautionary tale for the open-borders crowd!