Growing up in the suburbs of Chicago, I would have to agree with you... It’s not as if the majority of folks in the North at the time (or even now) were any “less racist” than those in the South. It wasn’t until the Civil War started losing “popularity” in the North that Lincoln started really pushing on the morality meme of slavery. Heck, at least in the South (in times past) you knew where you stood — in the North?? It’s all behind the back, and behind closed doors - it’s much more insidious, and frankly way more disgusting than those who proclaim their beliefs openly... I have no respect for anyone who bases a person’s worth upon the color of their skin, but even LESS respect for those who will smile to your face, and rip you to shreds when you’re not around...
Just to add support to your point, Lincoln didn’t free the slaves in the Union slave states, only the south. This shows the slave issue was a pretext. No doubt the slave issue was big for the south. But to say the war was about slavery alone is like saying the tea party movement is about health care.
And that wasn't always based on race, either...people of lower standing typically seemed to know it and there was a certain amount of order to it.
I'd prefer to have an unwritten understanding of who is who than one that pretends to be one but is the opposite.
Of course with the supposed ideal there would be no distinctions, but pretend as we might, there always will be; it is natural.