Then where do swing voters come from?
Let's say you usually vote Republican because you are rich and like lower taxes than you might get from the Democrats. Then they run a black guy for President and you say to yourself "Wow, that'd teach everybody we are not all racists" and you vote for the black guy.
You then spend the next four years on your knees demanding the Republicans do something about keeping the taxes down.
It works the other way. The Democrats started adding Jews to their "hated minority" list ~ and the Jews started jumping off the reservation ~ even voted for a Republican in Manhattan!
The Democrats are gradually backing off their anti-semitism. Probably not possible for them in the long run, but that ran a pretty fast cycle.