It was Captain Wirz, and conditions in prisons on both sides was deliberate. And reprehensible. Wirz deserved what he got, but he shouldn't have been there alone. There are others North and South who should have swung as well.
Most of the suffering at Andersonville came from the Yankees themselves. Wirz let the Yankees make their own rules and run the place internally.