The 1876 election is a special case but the black vote in the three Southern states still occupied by federal troops may have determined the outcome (making it possible for Hayes to contest the outcome and ultimately win when the 15-man commission voted on party lines to award him all 20 contested electoral votes).
The black vote started to be majority Democrat under FDR. Maybe that tipped the 1948 election to Truman and the 1960 election to Kennedy.
The first voters were freemen. White landowners. Not even all whites were given the vote and certainly not women of any color.