That's what a lot of people said about the early Mormons who practiced polygamy in accordance with their beliefs. In fact the federal government sent in troops to enforce other people's ideas about how Mormons ought to live. It would be nice to think that we'd moved beyond such self-righteous pig-headedness.
FWIW though, I think the government should get completely out of the marriage business, rather than expand its meddlesome regulation to include gay marriage. People's personal and/or religious activities are none of the government's business.
That is true until people's personal activities infringe on the rights of others and become a destructive force in society.