The big downside is not having a key with you when you need it. I keep one on my company badge retractor, one on my personal keychain, and one in a living room endtable. The keys are unlabeled, so if you lose one the finder/thief won't know who it belongs to (assuming you keep all identification odd your keychain). Without the physical key, I'm not getting into my systems (unless I remember the person). My PW manager generates very long and impossible to remember passwords for sites, too.
