I found out a few years ago that my Yankee relatives had bought and owned some of the very first black slaves sold in the US. To the best of my knowledge, mom's side were all small farmers and never owned slaves (but they did fight for the South in the Civil War).
But what matters is what "I" do....not what remote ancestors did more than three centuries ago.
The “owned” slaves surely lived far better than their ancestors in the old country.