I'm surprised to hear you say that. Especially since it has puzzled lots of people for a long time. The "billions and billions" argument--if you can call it that--breaks upon the pillar of the Fermi Paradox. It's like Olber's Paradox...which tells you that something in the received conventional wisdom isn't quite right.
The "billions and billions" coupled with the Drake Equation (with carefully selected parameters) says we should be overwhelmed with contacts. Rare Earth tosses dozens of 0.0001 probabilities at the "billions and billions" until you are down to some or none. And at the heart of it lies Fermi's famous question.
--Boris
I was rushed and it came out a bit harsher than I intended. :-(
I have looked at it and I still think the Lorentz Transformation is the key to why we are not being visited if anyone else is a tool building society that has achieved spaceflight.