Got a little personal anecdote on this - my last name is Whitehead, and when I served in the Navy, a lot of guys on my ship often shortened it to calling me "Whitey". The black guys usually didn't, and when one did inadvertently, he got all nervous and apologized. Then he got all fumble-mouthed and confused when we all laughed it off.
Could have been worse. They could have called you Hedy.
“It’s not Hedy, it’s Hedley.” Hedley Lamarr.