Gad!
Fear not, the beer starts warming up the minute you take it out of the fridge or once it's out of the keg. On a warmish day, most of the glass or bottle will be at around 45 (average) by the end of the first one, assuming you're not chugging it. If you start out at 36 or 38 degrees, it'll be nearer 50. That's just too warm.
You're right that beers taste more different at higher temps, but it's good to start cold. One exception: stout. Other than that, I say the colder the better.
You drink your beer, I'll drink mine.