You couldn’t just enslave anybody of color, and importation had been banned for decades. They would have had to bring in illegals or something to get labor. Honestly, even the south was not brave enough to give capitalism a chance - the idea that you could pay the former slaves to do their former work, and still make plenty of money - more money because your employees were motivated, not well understood.
If I own a slave then I have a life time commitment to their care. If I rent a slave (employ), my cost risks are variable.