If you don't keep a round chambered, it will cost you a couple of seconds and the use of your weak hand which may or may not present problems.
A firearm with a round chambered will go off in a fire, just as effectively as if someone flicked the safety and pulled the trigger. Further, unless one is already holding a firearm one will need to be able to get to it before one can use it. I would consider remote the odds that one would have 20 seconds to get to one's gun but not 21 seconds to get to it and ready it.
I would agree that it's foolish to pick up a gun for self-defense without immediately chambering a round. Among other things, if the gun is going to pick that moment to jam, it's far better if it happen when the badguy isn't on top of you than when he is.
The one disadvantage I can see to storing a firearm chamber-empty is that it reduces the magazine capacity by one. In most cases, though, I doubt that should be a problem. Even with a standard length unplugged magazine, one should have enough ammo to deal with any remotely reasonable situation.