This is exactly why I refused to provide anything other than the number of people living in my house and their ages during the last census.
They visited my house at least a dozen times to ask the rest of the questions on their list, and the “local chairman” left me countless messages threatening to “escalate” this issue if necessary.
They eventually gave up. I never gave in because I instinctively knew it could never be to my benefit to share information about my race or income level.
I did exactly as you did. I didn’t take too kindly to their threat to escalate either. Their notices were destroyed.