Of course another possibility is that a "colonization" took the form of genetic seeding of planets. In that case the answer to Fermi's question would be "You're looking at them".
Wasn't that a Star Trek Next Generation episode?