He refused exchange because the rebels didn't concider African American POW's equivient to redneck rebels. They got a taste of negro life, hey.
Many make the argument that the South fought the war to preserve slavery. If the South fought the war to preserve their slaves, then it was no wonder that they wouldn't return any of their slaves who were captured in Federal uniforms.
There was more to the North's refusal to exchange prisoners than the slave POW argument. For a time towards the end of the war, the North didn't want to free any Southern POWs they held, and they invented reasons not to exchange them because the Southerners would quickly return to the battlefield. Consider what the Union Commissioner of Prisoner Exchange, General Benjamin "Beast" Butler, said about blocking prisoner exchange after the war.
In case the Confederate authorities should yield to the argument...and formally notify me that their slaves captured in our uniform would be exchanged as other soldiers were, and that they were ready to return to us all our prisoners at Andersonville and elsewhere in exchange for theirs, I had determined, with the consent of the lieutenant-general [Grant], as a last resort, in order to prevent exchange, to demand that the outlawry against me should be formally reversed and apologized for before I would further negotiate the exchange of prisoners.
It may be remarked here that the rebels were willing enough to exchange prisoners at this time, man for man, were we to permit it to be done.