Did someone actually call you a racist for using that term?
I wouldn't say use of the term, 'colored', is racist, but it's long been out of favor among blacks.
The term that really grates on me is, 'African-American'. The only people who can honestly claim that title, are naturalized Americans who were born in Africa.
In the 70s if you called or said anything about a “colored” person the black guy would say “Who are colored people?”
It’s difficult trying to keep up with all the “approved” terms. Seems strange that you never hear the term “Negro” anymore.