You no doubt have many more enslaved and enslaving ancestors that you don't know about than that you do. The ancestors we know about are very few compared to those who are lost in pre-history.
I know 12 generations back and have been to the miserable villages where they served their andlords. How about you? Do you know who and where YOU came from? int no slave owning in my past. My ancestors were lucky to be able to share a pair of shoes to take turns herding sheep until they came to America expecting to find the streets paved with gold.
Not only was there no gold- but they had to pave the streets.