Even a SWAT team breaching a door fails to do so "instantly", even as they continuously train and strive to achieve such an ideal. If I'm three seconds or less from a gun I still like my odds against the typical home invading dirtbag. I have deadbolts locks, two doors on the house (they'd have to open a screen door, then kick or otherwise breach the wood door), a couple of large and alert dogs, and very little risk of home invasions in my neighborhood... there hasn't been one here in the last two decades that I've been able to discover.
So, again, if you feel it necessary to wear a gun 24/7, you have every right to do so. I don't believe that it is comfortable or healthy to live at such a "high alert" level at all times, especially when there's no reason to suspect that such a threat is very likely. Even as I type this I'm probably finding myself in my least armed condition... I'm sitting on my couch again (where I usually type these posts). My front door is about ten feet away from me. My gun is sitting on top of my night stand, just inside of my bedroom (where I left it last night). If some criminal decided to breach my door right now, I'm approximately 7 stride lengths away from my gun. I'd be out of sight of the door in two strides, and there is currently 160lbs worth of very home-protective dogs sitting between me and the front door.
Unless I'm invaded by a ninja, or a bunker-busting bomb, I still like my odds of being able to get hands on a weapon if I ever actually need to. In fairness to your point, a gun on the hip obviously saves a second or two compared to a gun on top of a table that is 15 feet away. But, again, life is about risk management, and I feel pretty darn secure right now (I'm also under no illusions about criminal activity -- I've been a ghetto street cop for quite a few years now, and I've seen quite a few home invasions where I work, most typically because of drug-related activities in the invaded home).