We did. Slavery is but a small and transient part of our history that was on its way being phased out by the industrial revolution.
The reason the South lost the war was because there was no manufacturing of great extent. The South, do in part to cheap slave labor never bothered with mechanization. The North out produced the South in guns ammo, shoes, etc.
So slavery was their undoing. Not only was it wrong but it built in a complacency that was resistant to change and modernization. I guess the slaves got their revenge after all.
A tiny portion of non African Americans benifited from slavery. Nearly all non African Americans have been paying the price for slavery over the last 50 years.