Certainly. Actively espousing the non-existence of God is just as much of a religious view as espousing God's existence. An atheist public official who wanted to put up a monument of his beliefs should also be prevented from doing so.
The best solution is for government to stay out of religion completely and we wouldn't be having these stupid fights.