In the south they don't seem to mind how close the blacks get, as long as they don't get to big.
In the north we don't care how big they get, as long as they don't get to close. -Tom
I believe that slavery was seen as not too economically viable at that time. This is (I believe) one reason why importation of slaves from Africa was made illegal in 1805. There was no one pushing for continued importation, because slavery was in decline. But then Eli Whitney invented the cotton gin, cotton became a huge cash crop and slavery was revived. I also believe that slavery was once again in economic decline in the 1850's and that many contemporary chroniclers (Frederick Law Olmsted, for one) predicted that the South would voluntarily end slavery in the near future. Perhaps this was wishful thinking -- but perhaps wager laborers might have made more sense than unmotivated slaves.
This article also mentions the curious fact that the Underground Railroad took slaves to Canada and points out that Blacks were not popular in the Northern US states. That may be true. But it should also be pointed out that the slavecatcher laws allowed escaped slaves to be dragged back down South if found in a northern state. Under such circumstances, Canada was the safer place to live.