The answer is, that special charisms of this sort are given for the benefit of the Church (the whole Church or part of it), not the individual -- just as being understood by foreigners of another tongue, or just as prophecy (private revelation) also may be. The charism does not depend upon the individual (except they be disposed to receive it) but the needs of the Church.
The Holy Spirit responds to the need of the Church for celibate priests by manifesting the Gift of Celibacy within individuals, so they may answer their vocation and live it faithfully, for the greater glory of God.
(Testosterone levels are irrelevant -- apropos a different poster.)