Actually, Fermi did have a "clue", which is that we don't see any evidence of other advanced civilizations in our galaxy. Fermi wasn't concerned with bacteria - you want the Drake Equation, see earlier posts. I actually think life will be widespread - but nothing much more advanced than an amoeba. I don't have the stats to hand, but I'm pretty sure that life on earth consisted of single cell organisms for 90% of the time.
I understand, but for all he knows, there could be planets out there teaming with two headed baboons or human like creatures that have no desire to build a space craft and travel to the stars, or a race of creatures with little intelligence, (like those in DC) or maybe these creatures life span is only 10 years, and they don't have the mental ability for space travel, or it's taboo or they just don't have the natural resources to build a craft etc... Hehehe.
I for one, believe that there is probably a whole lot of life, right within our very own Milky Way galaxy. We just don't know what is going on the other side of the galaxy, say 80,000 light years aways, on the surface of some small planet. Not yet anyway.