Not logical. A vastly larger percentage of Americans are descended from slaveowners, including Dear Leader and myself. But then I'm also descended from men who fought to end slavery.
More importantly, that 385,000 people owned slaves in 1860 is true, but very misleading when you use the number to determine the number benefitting from the institution.
As with real estate today, ownership is generally vested in the head of the house, but we don't say the family doesn't own their own home.
In the Lower South, 40% of white families owned slaves, in the Upper South 25%, and about 10% in the Border State. I'm working from memory, so numbers may be a little off.
Your number only makes sense if we agree Scarlett O'Hara had "nothing to do with slavery."