That's not entirely true. Many of the founders were slave owners through inheritence but were repulsed by slavery and fought it. One of them set his slaves free which resulted in laws being passed that prevented the others from freeing theirs.
Slavery was not a ... pardon the pun... black and white issue.
In Alabama, Mississippi, South Carolina and many other states down there, it was certainly a black and white issue. If that were not the case, then blacks would not have left in droves to go to the safety of the North.