Because the term is used for the purpose of extortion against a minority that thrives as victims.
My reaction is immediate and aggressive. Attack back. “Racist” is the worst insult on the street. They win when you boast of black friend, relatives and good works.