I'm kind of surprised they wouldn't have torn it down or at least gutted it a long time ago since supporting Nazi ideals is very illegal in Germany.
Since the nazis erected many buildings there are quite a lot obtained. Sometimes some insignia was taken out. Espechially many barracks that were used later by the US-forces are a leftover from the nazis. Most of them had a "Pleitegeier" ("Hitler's vulture" - the eagle-emblem of the nazis) on their façades. Since your compatriots seem to have a obsession to eagles they often simply chiseled the swastikas away. ;-)