What’s the alternative? I agree with you that pastors have a religious authority to perform marriages, but say you have a believer who marries another within your congregation. Then it’s found out later on that the fellow is already married. Part of the reason why they are legally binding marriages is to protect both parties from fraud.
How are you to confirm that he’s already married? If your argument is that only religious marriages are valid, you will get people taking advantage of that provision to get married even though they are already married.
I understand your frustration, up here we have pastors and priests who get leaned on to perform gay marriages, and public outcry and newspaper articles when they say NO!
How do you suppose people managed back in the 1700s and early to mid 1800s, when many marriages in the US (maybe the majority, especially in thinly populated frontier states) were common law, and may not even have had a ceremony by an itinerant preacher, much less a government license? And do you seriously think anybody, either in the government office issuing the license, or in the religious entity performing a religious marriage ceremony, is running a national search to find out if either of the proposed parties is already married? Of course not. That's why cases are discovered every year of someone who's simultaneously married to multiple partners, without those partners' knowledge of the others (usually for some sort of financial gain).
The government has managed to brainwash practically everyone into thinking they just can't manage their own lives without government involvement in virtually every aspect. It just isn't true, and it's a dangerous mentality. If government wasn't running this marriage business, then people wouldn't be relying on a marriage license to assure themselves that they'll inherit their spouse's assets -- they'd make sure there's a valid will. And people wouldn't be forced to leave their hard-earned Social Security survivor benefits to a lying, sleeping-around, drunken, profligate-spending spouse who they stayed married to for reasons of children or religion, or because the creep couldn't be located for a divorce proceeding, instead of to their saintly younger sibling who's been caring for their Alzheimer's-stricken parents for the last 10 years.
If government isn't involved in marriage regulation/licensing and someone wants to get married, then they should decide whether they want to do this as a private contract or through a contract approved and recognized by a religious denomination they belong to. Presumably, if the parties aren't consenting to polygamy or polyandry, one of the terms of the contract would be that both parties represent they are not parties to any similar contract with another living person, and that entering into such a contract with another living person would constitute a breach of this contract with specified remedies. Religious denominations would draft standard contracts that agree with the teachings of the religion, perhaps with some room for optional provisions to be chosen by the parties to the contract if that's consistent with the religion's teachings, and if people chose to enter into that contract, it would be binding like any other contract.
Look at the mess created by government-recognized marriage when multiple states are involved. Say two people get married in one state -- there's no requirement that they have any clue what the financial ramifications are per that state's laws, but for the sake of argument, let's say they do and they know it's a community property state. Wife works full time in some boring dead-end job that's all she's qualified for, to put husband through medical school, knowing that she's entitled to half the future earnings from the medical degree, even her husband later decides to dump her and run off with some cute young nurse. Fourth year of medical school comes along, and husband has to participate in the federal government-mandated National Match program which determines where he will spend 4 years in a residency program (if he doesn't, his degree will be rendered virtually worthless by the government, since he won't be eligible for a medical license, insurance reimbursements for services, etc). He's assigned to a great program in another state and everything is still going hunky-dory in the marriage. Wife is thrilled, and off they go the other state, with wife either not knowing or no longer thinking she needs to worry about whether it is or isn't a community property state. It's not. Three and a half years later, just as the insane and unhealthy work schedule and appallingly low (and non-negotiable) pay is coming to an end, and the life of a financially comfortable family with a full-fledged medical doctor as primary breadwinner is about to begin, husband dumps wife and runs off with a cute nurse. Oops! Wife's not entitled to half his future earnings anymore -- they're well-established as residents of a non-community property state now. When did she agree to this massive change in the financial underpinnings of their marriage? She didn't. Doesn't matter. The government decides these things for people. One state says "marriage" means one thing and another state says "marriage" says another thing, and any state can change its own definition any time it likes, without the consent of all the people affected by it. Does this sound like freedom to you?
Free citizens really CAN manage their own lives. Don't let the nanny-staters convince you otherwise.