So, whenever I picture myself wearing a concealed weapon, I invariably picture a handgun in a holster on the small of my back..."
-Small-of-the-back is a dreadfully inconvenient location, sometimes hard to get at, and very uncomfortable in vehicles or furniture. Examples of spinal injuries caused by falling on the gun also exist. Detective novels and thrillers seem to love the SOB position. In the real world, forget it.
" I think a gun on my hip would tend to be noticed, as that's where I also keep my pocket knife in it's belt pouch, not to mention that it might get exposed while I reach into my hip pocket for my keys."
-For heaven's sake, if you intend to change your lifestyle to the extent of carrying a hidden weapon, you should be prepared to MODIFY the way you carry the less-important impedimenta we all tote around. MOVE the knife! Carry the keys SOMEWHERE ELSE! For fifty years, I carried my wallet on my left hip, because I'm left-handed. When I moved to a place where I could get a carry permit, I made the conscious decision to MOVE the wallet to my right hip, for exactly the reason you mentioned....not wanting to expose the gun on my left side when reaching for the wallet. My knife also got shifted from left-side to right-side.
Concealed carry means making adjustments, in how we act, how we dress, and how we think. The gun and its secrecy, concealment, comfort and availability MUST become THE priority, and all else is secondary.
Here in semi-rural Nevada, the unbuttoned shirt over a T-shirt is almost a uniform for much of the male populace in hot weather. We also have a lot of "breezes" (meaning real WIND), so keeping the shirt-tails down is sometimes a problem. Mostly I just press my arm against my side to hold the shirt until out of the wind, but I know some folks who had their wimminfolks sew a short length of light chain into the hemlines of their "carry shirts" to weight them down. As many others have mentioned, that loose outer shirt can conceal a BIG gun. Mind you, I've also carried an N-frame S&W .44 Magnum under just an un-tucked T-shirt on a few occasions, and no one noticed.
You're getting good advice from the folks here. Best of good fortune to you, and stay safe.