Do we have the right to not answer questions concerning the census if the person who comes to our door is an ACORN employee?
The the fact that the person is from ACORN is irrelevant.
Anyway, I assume (not a legal advice) that nobody has to answer questions when somebody shows up at your house. Freedom of speech covers silence, I assume.