Black people owned slaves too. During the War Between the States, the third largest holder of slaves in South Carolina was black former slave.
Should we despise blacks too? (rhetorical)
Indians owned black slaves, too.