The "men can't be expected to live this way; let's bend the rules" argument is the same one made by the early Mormons on behalf of polygamy.
Check out the numbers on adultery among Protestant clergy. It ain't pretty.
Which sin is greater to commit adultery against a wife or commit idolatry against the Heavenly Father. IICorinthians 11:2-4
Which rules are those? It seems the heart of the debate whether God actually has a rule that men are better off celibate, and that marriage is a lower form of existance.
We can argue that point, but your summary begs the question, assuming the rightness of your side.
Of course, it was Paul who made the argument you seem to ridicule.
My grandfather was the only maritally chaste minister his Southern Baptist Church had during the 20 years from 1980 to 2000. All the rest of them had to be run out of town for participating in horizontal counseling of the forlorn ladies of the parish on the Pastor's office couch. From reading the newspapers down south of these sorts of goings on, and also noting the enormous number of divorces, remarriages and affairs reported among southern evangelicals, his parish was not atypical at all.