Unfortunately, this is frequently true. However, it is not a doctrine of the Church that we should behave this way. In fact, it has been the subject of more than one formal address by the President of the Church. He regularly encourages us to be more inclusive in our social lives.
In my opinion, the problem stems from the fact that many (most?) people of other faiths, or of no particular faith, engage in recreational activities that are either uncomfortable or forbidden to a practicing Latter-day Saint. We aren't going to join our drinking friends at the bar, for example. Others may spend Sundays, observed as the Sabbath by the Church, without their LDS friends, water skiing or fishing. Since recreational time is one of the ways friendships are maintained, this requires more of an effort and a conscious decision on the part of these friends to find activities acceptable to both.
For the vast majority of Mormons, at least those I know, this situation isn't a matter of "holier than thou" per se, but a simple conflict of values and beliefs.
Hope that wasn't too much of a "sip from the fire hydrant", but I know it's a sore spot for some when they think of the Church.