One more thing...Modesty is almost always discussed in terms of how women should dress, but I know for a fact that if I appeared shirtless in public, no woman would be able to resist having inappropriate thoughts about the twisted-steel-and-sex-appeal on display.
Modesty is not relative, but it is very situational.
A bikini in church is very immodest. A regular biking on a beach full of women in string bikinis could be quite modest.
(I have a similar problem to you as to going shirtless. I try to never go shirtless on the beach as people keep trying to roll me back into the ocean!)