Not to worry. I work in an environment that is about 70/30 men to women. The ladies are just fine. Hard working and fit into the office banter without problems. On the rare occasions a complaint was raised, it is almost always caused by heightened sensitivity due to issues causing problems outside the office. Overall problems are very, very rare.
Your guard is down and you need to be careful. They are like wild animals. Never turn your back or or get comfortable.