Because they were Southron patriots, and because they didn't cotton to a Yankee Army invading their homes and ripping up their institutions?
Any white southern non-slaveholder had a vital interest in containing the violent and potentially violent African population which lived among them. The South did not practice Northern-style segregation. Africans were, at least in the Deep South, everywhere.
The North never gave, and has never given to this day, a coherent theory of race relations (a/k/a "diversity") which could persuade any sentient white Southerner exactly how what would happen AFTER emancipation would or even could be good for him and his family.
The southern white laboring non-slaveholder had every reason to fight.
Guess I'm just a soft headed liberal in that respect....