Maybe people just don't trust a guy in a dress?
We had a guy in a dress here about ten years ago. He was 6' and 225lb. Two of the girls in Acct., both about 4'9" went into the ladies room and he's standing in the stall with the doors open. They were horrified.