The first voters were freemen. White landowners. Not even all whites were given the vote and certainly not women of any color.
Some states in the early Republic allowed free black men to vote--even a slave state like Tennessee (until it adopted a new constitution in the 1830s). New Jersey let some women vote in the early 1800s (I think they may have had to own property). That did not last too long--maybe a couple of decades.
Alexis de Tocqueville was in Philadelphia on election day and noticed that no black men were coming to the polls--he was told they had the right to vote (but were afraid to exercise that right).