These would be problems to us, but not necessarily to all cultures. All that would be needed for a galactic empire would be one race willing to put up with these difficulties and enough time. As Fermi points out, time isn't a problem. Were there enough spacefaring races, then at least one of them would have had the patience to build an empire by now.
On a side note, sci-fi author Harry Turtledove addressed the issue of patience in his "Worldwar" series. In those books, the Earth is invaded by "The Race" during WWII. The Race has been a spacefaring people for thousands of years. Their plans to invade Earth evolved over centuries. They sent a probe to Earth that arrived 600 years earlier. They were so confident that all races shared their patience for slow development that they expected to find nothing more than mounted knights opposing their invasion force; not modern armies able to fight them to a standstill. One of the major themes of the books is the Race's inability to grasp how impatient we are as a species and how this drives our technological development. The Humans, on the other hand, learn patience from the invaders and have begun to plot a counterattack on the Race's homeworld, Tau Ceti.
Here's what I think:
The limits to empire are not willpower and climate. The true limits are economic. Look at the USSR. What finally did them in was economics. All those resources were there in Siberia, and still are, but distances threw the economic equations out of balance. Even the train system couldn't balance the annual report. Extend the economic equations to interstellar distances and see how willpower has any effect at all.