The temporary exhibit at the Perryville Battlefield Museum I mentioned to some others in comment #115 also has a detailed look at the guerilla fighting that took place here in Kentucky, including a pistol carried by "Sue Mundy", the infamous Confederate guerilla.
A number of these outlaws did not fight for either side of the war, but merely used it as an excuse to rob, pillage, and murder.
Most the men who fought for both sides who brave men who honorably did what their conscience told them. But both sides had their bad element. And both bad elements tended to make themselves scarce when real fighting was to be done.