I did some work at a government facility that had all sorts of barn swallows. This one guy, every morning, would go around with his broom and sweep the overhangs of all the buildings.
When I asked him what he was doing, he said he was knocking down all the bunches of twigs and stuff that the swallows were putting up. If he knocked them down every day, they never got big enough to become nests! As you said - it was illegal for him to knock down a nest.