Next time he contacts you, ask for the name and number of the director of his department. Tell him if he doesn't give it to you, you will go to the press and detail the harrassment you've been experiencing.
The press isn't exactly going to fall all over itself to break the stunning "Census worker leaves NOTES ON MY DOOR!" harassment story.