Interesting issue--one that I haven't seen before--but I haven't met many libertarians who believe it's ok for one man to own another man.
Have you met any libertarians who would favor invading another country to end slavery?
The freedom loving, libertarian Southerners were enslaved by the Northern push for the end of slavery. Is that about it?
Back then, it was "okay" and indeed part of a person's individual property rights, which a Libertarian would be bound to defend. Disestablishing slavery, then, would require a new social compact freely entered into.
That wasn't what happened.
Property rights in other people were ancient and had a Biblical sanction unrebuked by the God of Abraham: Abraham himself sired Ishmael on Hagar, the bondwoman of Sarah Abraham's wife (and the siring was Sarah's idea). Hagar was a woman of Egypt, serving out her life in bondage in the land of Israel.
One of the oldest documents in sub-Roman France, written in what is now called the "Lingua Rustica Romana", is part of the documentation of the emergence of Old French. The document is a bill of sale for a slave and reads:
"Constat nus ut aliquom fimenom nomine Nautlindho vindemus tibi pro pecia de ma[n]so probrio jures meo."
The man is selling a woman named Nautlind (a German name) for cash and specifically cites his "personal property rights" as the source of the sale.
Giving up a right of high antiquity for the reconsidered public welfare interest in a sounder basis for a compacted society would not have been an unreasonable thing to do, if rights had not been ruptured by acts of war that did violence on, and did away with, the old Republic, substituting a barely-concealed Empire in its stead.