Indians and blacks owned slaves, IIRC.
I've read that the first man to be enslaved indefinitely (as opposed to entering indentured servitude for a time period, which was common) was owned by a black man in Virginia. It came about as a result of a court decision in his favor. The (black) owner had sued a man who was trying to win his freedom. As I read the record, the court agreed with the owner and laid down the precedent for lifetime servitude.