First of all no one has the right to come on your property without your allowing them to. Secondly you should have asked him to show you identification, asked for the name of his supervisor and a phone number that said person could be reached at, told him to go wait in his car while you made the call. And then without this ‘’census taker’’ being aware of it you should have called the cops and told them you have someone on your property posing as a census taker and you want them to get here right now.
There are people who are allowed on your property, even with the “no trespassing” signs. And yes, if you tell them to leave, they have to leave. But they can come back. And no, I’m not talking about the police.
And you all know that with a “no trespassing” sign, first you have to tell them to leave. If they return (above exception), THEN they are trespassing and you should call the police or take whatever action you feel is responsible, always with the law in mind.