To be fair, the North wasn’t trying to crack down on Southern slavery. Not while it had the same institution itself.
It still was (to use a metaphor) bad karma. If someone can harden their heart badly enough to support that, they are going to overlook a mountain of other blessings too.
You’re correct. The north was vastly more conflicted about slavery than the south. Opinions were all over the map. In the south however things were monolithic - the powers that be were going to have slavery and would willingly defy God himself to keep it.