You’re not a rape victim, or someone else with reasonable anxieties in the subject.
Segregated washrooms were created by most societies for good reason. Just because that’s working pretty well in preventing problems doesn’t mean we should facilitate those problems thru legislation. Tolerating exceptions and minor transgressions is fine; forbidding averting the onset of a serious problem isn’t.
Most rapes occur outside public bathrooms. I’m hard pressed to think of a case where a rape occurred in one. I’m sure there are such cases but how would a different law have prevented it?
Law abiding people like us sometimes forget that criminals have no regard for the law. I don’t think we can legislate away crime, as much as we like to think it is that easy.