I guess we should ban the USA flag as well.
There was not a single solitary slave under the flag of the confederacy who did not begin as a slave under the USA flag. Every one of those confederate slaves who resulted from secession all began as US slaves on US soil.
So really, we should ban the Stars and Stripes as well, since slavery was begun and was legalized under the Stars and Stripes.
If it is good for the goose, it is good for the gander.
There are those calling for that even as we speak.