I promise you that a sidewalk cafe with temps in the "triple digits" is better than one with the temps in the "single digits" or slightly warmer.
You have a point, although I found I was always able to add enough layers to stay warm, but you can never shed enough layers to stay cool. Although it is hard to handle a fork when you have mittens on over your gloves.