“No refuge could save the hireling and slave
From the terror of flight, or the gloom of the grave”
I’m guessing this is the section they object to based on the word slave. What morons. In classical times a slave was someone who was a subject, servant, or bondsman/woman. In this context I think Key is talking about the British being slaves because they are subject to the King of England. It is intended as an insult to the English, because they are not citizens of a free Republic like Americans are.
But only an ignorant modern-day moron would object to this thinking it is an aspersion against African Americans.