That rather sums it up I'd say...
1Tim; 3:5 "If anyone does not know how to manage a family, how can he take care of God's church?"
This verse refers to bishops that were widowers. Paul is instructing that these widowers could not remarry. The verse also refers to those bishops who were currently married. They also could not remarry (in the Catholic Church's Eastern rite, priests are allowed to marry; celibacy is only a disciplinary rule for the clergy of the Roman rite). Therefore, this text has nothing to do with imposing a marriage requirement on becoming a bishop.
On the other hand, the number of Scriptural verses that support priestly celibacy is far greater.
"But I would have you without carefulness. He that is unmarried careth for the things that belong to the Lord, how he may please the Lord: But he that is married careth for the things that are of the world, how he may please [his] wife. There is difference [also] between a wife and a virgin. The unmarried woman careth for the things of the Lord, that she may be holy both in body and in spirit: but she that is married careth for the things of the world, how she may please [her] husband. And this I speak for your own profit; not that I may cast a snare upon you, but for that which is comely, and that ye may attend upon the Lord without distraction." - 1 Corinthians 7:32-35
"Now concerning the things whereof ye wrote unto me: [It is] good for a man not to touch a woman." - 1 Corinithians 7:1
"And every one that hath forsaken houses, or brethren, or sisters, or father, or mother, or wife, or children, or lands, for my name's sake, shall receive an hundredfold, and shall inherit everlasting life." - Matthew 19:29
"His disciples say unto him, If the case of the man be so with [his] wife, it is not good to marry. But he said unto them, All [men] cannot receive this saying, save [they] to whom it is given. For there are some eunuchs, which were so born from [their] mother's womb: and there are some eunuchs, which were made eunuchs of men: and there be eunuchs, which have made themselves eunuchs for the kingdom of heaven's sake. He that is able to receive [it], let him receive [it]." - Matthew 19:10-12
Emphasis mine.
Finally, to put it bluntly, one doesn't need to be an adulterer to counsel other adulterers. Priests understand the sacrificial nature and sanctity of marriage in a way that few others do. Who better to counsel a person in the ways of keeping the marital vow of fidelity than one who keeps the vow of celibacy?