They are not really US citizens and have no allegiance to America.
Some ARE U.S. citizens free to travel about the country.
How are you going to identify and eject them? It’s not like they carry an ID card or tattoo or something.