The North fought to preserve the Union (and tariffs btw) while the South fought for free and independent states. Slavery was an issue but not the only issue. The quotes and actions of Southerners, including generals and commoners are peppered with pro slavery diction. But if you look long enough you'll also find anti slavers among those dedicated to fighting the North (Lee, Hampton). If slavery was such an issue then those leaders who welcomed the end of it would have been pariahs and such statements would have been blasphemy.
Sorta like me saying "well, Ted Kennedy isn't so bad".
And while you might be correct that there were only a handful of hardcore abolitionists, there was a widespread distaste for slavery and a desire to see it contained. The Republican Party's 1860 platform talked extensively about slavery. Their electoral success in the northern and midwestern states gives lie to the idea that northerners didn't care about it. The further fact that the south was endlessly complaining about northern opposition to slavery, specifically in failing to (in their minds) enforce fugitive slave laws, must necessarily mean that there was opposition to slavery and, more specifically, a dislike of being compelled to be complicit in maintaining it.
Can you really say that Lee was anti-slavery? And Hampton? Isn't it a question of not being enthusiastic about slavery as some were? Or perhaps of taking doubts about slavery or secession to mean actual opposition to slavery?
And slavery btw.