The "Confederate Flag" you speak of was not the flag of the political entity called the Confederate States of America. That was the "Stars and Bars". The flag we are discussing was a Battle Flag.
The swastika was a political symbol of the Nazi Party. The Iron Cross, on the other hand, was a battle emblem that is still the battle emblem on German aircraft and armor to this day although the Iron Cross has also been misused by some groups.
If all it takes for a symbol to be banned is it's misuse by certain hate groups, do you think the flag illustrated below should be banned?