I don't agree with that at all. Why would it be rude for two people on the street to conduct their conversation in Japanese, for example?
I think this applies to business settings.
That would a "private" not public conversation. What I mean, for instance, I was at the the hot tub at an apartment complex the other day, there were these two young girls (I think are from Korea, I think are here on student visa, i.e. visitors). They both speak English as evidenced by the one girl speaking to us and we speaking to her, the other girl said about one sentence in perfect English and the rest of the time, only spoke to the other girl in Korean while the rest of us included the polite, young lady in our conversations. We did not however, include the other young lady. She was being rude by not speaking English (no telling what she was saying about us). She deliberately spoke her foreign language in full knowledge that no one else could understand her. That is what I mean by "rude".