Seems to me that the phrase "three out of four of us" is quite plainly incorrect, as you say there were five of you at lunch.
I am not surprised you were audited. Sounds like you got your audit the old fashioned way.....you earned it.
My mistake. I was one of the four.