Technically, yes, it’s mandatory to reply, in terms of statute you can be fined for not replying, but legal action is never actually taken against individuals and nobody actually gets fined.
However, if you don’t reply you’ll constantly have a variety of Census workers calling/visiting for a couple of months.
Hard to believe this would have been started by President Bush.
Why would anyone care what time I leave for work in the morning? Except perhaps someone wanting to know when the house was empty for nefarious reasons.