I have yet to encounter one single opponent of this mosque who says the Muslims do not have the right to build the mosque at Ground Zero or wherever they please.
The issue is not can they do it; the issue is, is it a good idea? Will it serve to "build bridges" between the faiths? The verdict is in -- look at the damage it's done already.
I would also add, their constitutional rights do not give them immunity from criticism when they choose to exercise those rights in a destructive way.
What was the first comment I said ?