Some 15 years or so, I met a distant cousin on an internet genealogy site. We never met face to face, but we exchanged emails for several years. His family came from North Carolina and arrived in Brazil with no money, but build a good life over several generations. They were not wealthy and very few ever went to college. In many ways their standard of living was not changed beyond that they knew in North Carolina. My guess was that they had lost about 3 generations of economy progress by moving to Brazil. This guy spoke only Portuguese, although he could write fairly well in English.
C-SPAN3 has been running some interesting programs having to do with the end of the Civil War on recent weekends. One segment dealt with the Confederados. It seems about half of them eventually returned to the US. Some of them had it pretty rough in Brazil—one speaker read something from one of the settlers (either letters or a diary, I forget) describing his life there the first few years, a very difficult life.