This is NOT true. How much "over time" (meaning history) does one want to go back through? There are nearly 8 billion people alive today, but there were an estimated 110 billion people who lived throughout human history prior to our time. Most of them had hard-scrabble lives, but generation by generation they built the modern world. Civilization, as we understand it, began among light-skinned people living around Mesopotamia, the Mediterranean and Europe.
If one only wants to focus on American history from colonial times, European immigrants built the country, not by privilege, but through hard work.
Don't leave out the POC in the Indian subcontinent and China/East Asia.
Based on the impressive achievements of the various Mesoamerican cultures and the Inca, the Native Americans might have gotten there in time. Mesoamerica had writing and was working gold and silver, but hadn't made the leap to metal weapons and tools. That's a developmental curiosity.
Then there are the parts of the world that neither developed writing nor developed beyond the Stone Age. The descendants of these cultures are what the current fuss is mostly about.