Very valid observation...No different to have a heterosexual leaning priest to move in with a bunch of nuns; even using the same restroom facilities as the nuns...Torture the celibate priest with temptation...
This does create difficulties for same-sex-attracted men who are trying to achieve the inward peace of chaste singleness for the Lord. With men, they tend to develop crushes, uncomfortable stirred-up feelings. But they can't normally just go and hang out with women; nor are they necessarily suited for a starkly solitary (hermit-like) existence.
The SSA men I have known who are trying to walk in chastity, have had quite difficult struggles. Really, they need our prayers and our understanding support.
The Catholic chastity support group Courage has this as its aim; unfortunately, they remain a small fledgling group.