Spot On. I am SO tired of people going into a mental fetal position wherever "racist!" is hurled at then and plaintively whining "But I'm NOT a racist". Twice I've had that used at me, and when I replied, "Come up with something else please, that old slur is SO pre 9-11" (thank you Victor Hansen). I get the deer-in-the-headlights look. I think it's partially because I don't fold, but also because I'm confrontational. Many have noticed how these professional victims fold at the least sign of resistance. It's something we ought to practice more often.
One of these days I'm going to say "So what? Waddaya gonna do about it?" and enjoy the back-pedaling - or get the satisfaction of returning a punch.
I do that now. It stops them dead.