I fly in shorts because it's more comfortable. Why be cramped in a too-small seat, in a hot, stagnant airplane, AND still be wearing very uncomfortable clothes in the process?
Wear what you will, and I'll bring my kids. Oh but you should buy new clothes if they are uncomfortable. The sports coat carries passports, travelers checks and other papers, is great in a knife fight, and much needed at a good restaurant. I can go to Europe for two weeks in a medium sized suitcase, partially because I wear the coat instead of packing it.