Agreed, totally. Even if the galaxy is uninhabited by other civilizations, when we finally get off this mudball and get out there, we will have to break up into separate self-rule entities. While we might spread out throughout the galaxy in a million years once we get moving, linkages to a central authority would be next to impossible.
These would be problems to us, but not necessarily to all cultures. All that would be needed for a galactic empire would be one race willing to put up with these difficulties and enough time. As Fermi points out, time isn't a problem. Were there enough spacefaring races, then at least one of them would have had the patience to build an empire by now.
On a side note, sci-fi author Harry Turtledove addressed the issue of patience in his "Worldwar" series. In those books, the Earth is invaded by "The Race" during WWII. The Race has been a spacefaring people for thousands of years. Their plans to invade Earth evolved over centuries. They sent a probe to Earth that arrived 600 years earlier. They were so confident that all races shared their patience for slow development that they expected to find nothing more than mounted knights opposing their invasion force; not modern armies able to fight them to a standstill. One of the major themes of the books is the Race's inability to grasp how impatient we are as a species and how this drives our technological development. The Humans, on the other hand, learn patience from the invaders and have begun to plot a counterattack on the Race's homeworld, Tau Ceti.