The Catholic Church was famously described by Joyce as "Here comes everybody". The Church has all sorts, including but not limited to, "timid, retiring intellectuals" (oh, the horror!), as well as perverts (which is fine as long as they recognize perverted activity as a sin and strive to overcome it). If you are talking about the priesthood, I think that the criteria have been tightened up in recent years for the better (at least on the practice and advocacy of perversion front).
I think you underestimate the negative effect the timid, retiring intellectuals have on the image of the church. The boy who never went to a high school dance and now seeks the comforts of the robe in order to escape having to face his sexual demons or support his own livelihood ... not fair, agreed, but it is the image nonetheless. Remember, to outsiders, the Priests and Nuns are the Church.