The real problem is that they molest children.
All the evidence I have seen suggests otherwise. Sure, some child molestation happens, but the largest majority are not children (defining pre-pubescent as children). Most incidents involve post-pubescent youth or young men.