The only effective limitation on this was the initial US Army field retaliation, which involved about 20-40 expedient hangings of newly captured Confederate personnel. This retaliation ceased immediately on Lincoln's orders, but pretty much deterred further atrocities by CSA personnel against colored troops (they were instead enslaved), excepting Forrest's who continued to murder quite a few captured colored troops for the rest of the war.
Civil wars are always ugly. The Confederates would not surrender and had to be destroyed inch by bloody inch. Tough for them. The North was outright nice compared to what Europeans would have done in the face of a refusal to surrender.
And the South didn't try guerrilla warfare only because they knew how the North would use colored troops.
Actually the South used guerrilla tactics quite a bit. Stand Watie a Cherokee General in the Confederate Army used them effectively. Probably the most successful partisan Ranger ever was Confederate John Mosby. My Mother’s GGsomething belonged to Laird’s Rangers and she had another kinsman serve in the Alabama Partisan Rangers.
George Custer started the hanging of prisoners but John Mosby made him think again. Custer captured a small number of Mosby’s men and hanged them. Mosby who had always just released his prisoners via parole and promise they would not fight again, went out and captured a whole bunch of Custers men and handed 10 for every one Custer had done the same to.
That ended the murder of prisoners by Custer.
You have it exactly backwards. It was the Union which began the murder of prisoners and the Confederates who put a stop to it since they had far more Union prisoners than vice versa, especially early in the war.
First, Forrest tried to stop the Fort Pillow massacre. That’s based on Union testimony. Second, the Fort Pillow massacre pales in comparison to Union atrocities in the Shenandoah Valley and Georgia. Third, why should the south have “surrendered”? The states of the Confederacy exercised their right to leave the United States. Fourth, it was not a “civil war”. Check the definition. Lincoln systematically violated the Constitution.
At Andersonville the black troops were treated better than were the Tennessee Unionists according to the first hand account book, Life In Rebel Prisons written and published in 1865.