Too bad those guys hadn't got in touch with Confederate big shot Howell Cobb who said:
"if slaves seem good soldiers, then our whole theory of slavery is wrong."
Sherman and Grant thought very very poorly of blacks as soldiers and would not let any fight in their armies. But yet blacks served in other parts of the union armies (albeit they had to buy their own uniforms and were paid less).
Numerous blacks served in the Southern armies too, regardless of what a few folks may have thought about it. Many southern generals (like Lee) were in favor of having blacks in the southern armies. Get over it.