Precisely. To vote, one had to be free, white, male, 21, and a landowner. There were large numbers of people who were counted as population who were not eligible to vote.
We do need to eliminate the illegal aliens from the equation, though. They are not supposed to be here in the first place.
The way to eliminate illegal aliens from the equation is to simply deport them. Then this whole issue becomes nothing more than an academic discussion.