But handguns in particular need to almost always be ready to use. An unloaded gun used to make a decent bludgeon, but the newer ones are so light, that is not as true as it once was. ( I still have a polymer framed pistol, but it's loaded, as are the all steel 1911, and magazines for the so called "assault weapons".).
No sense in having it for defense if it isn't chambered IMO.
Personally, I am more comfortable "cocked and locked" with a Colt than I am with a trigger safety Glock, however that is just me.
Either are safe with a competent carrier and with me I am just as fast with a Colt with an external safety as many who tote a Glock with a trigger safety.