There have been others. Just not from the U.S.
The 1936 Olympics were awarded to Berlin 2 years before Hitler and the Nazis came to power.
“Jesse Owens’s participation in the Olympics was controversial because of his race, at a time when segregation and discrimination against blacks were the norm in much of the United States. However, once in Berlin, Owens was able to freely use public transportation and enter bars and other public facilities without the difficulty he would face as a black man in the United States.....”