Unfortunately, I don't trust the frangible rounds or bird shot. I have to have confidence in the defensive round.
Totally agree here.
First off, basically anything that will do the job will go through thin walls. Just know that. 00 buck can overpenetrate, as can the .45 and 9mm. Those are all heavy loads with lots of momentum. What you need is a lot of kinetic energy. Light and fast. But were I to go with a handgun, I'd just use either of those, but I'd use a 115gr. in the 9mm, I'd use a Winchester +P+ because I have them, or the 200gr. Gold Dot in the .45. It opens up so big I can't imagine it having much energy left if it did overpenetrate. For what it's worth, I keep my carry pistol next to the bed too, either a .40 or 10mm. But I just like to keep it handy, I have something else for HD.
My vote is with the M4. Mine is a military grade rifle, Noveske lower and all mil spec parts, with an LMT 14.5" bbl. upper pinned with an AAC flash hider for a suppressor (I don't use it for HD though). I just built it and it shoots great, so I'm kind of proud of it if you can't tell. Anyway, this makes for a nice, light, short and easy handling rifle that is accurate, fast, and low recoil. Feels nice to me since it is a copy of an army issue rifle, KAC rail, broomhandle, Surefire light, and ACOG reflex sight, just like I had in the army but with a Geissele 3gun trigger instead of a burst trigger (also no ACOG scope or bipod, don't need it). It never let me down and it just feels right in my hands. I've done so many shoot houses and clearing drills, used one, carried one, and shot one so much, that using anything esle would be awkward. Also, using a reflex sight on a short rifle like this is super accurate at close range, it shoots fast, it is almost like cheating. A shotgun is a good HD weapon, but it has a lot of recoil, slow to bear and shoot, it is the reason our entry teams didn't use shotguns except to breach. The M4 was the preferred tool for CQB, house to house. Why would you choose anything else for your own house?
I don't have shared walls like you do, my neighbor is kind of close, but we have a lot of trees between us too, as well as an outbuilding connected to a carport --the round has to go through a lot to be a menace, in other words. He uses a full length AR, his competition rifle, for HD. I have no worries about it at all.
The 5.56mm round will usually not overpenetrate, and if it does, it is usually fragmented. The high velocity lower weight rounds just come apart. The 55gr. XM193 is a good load for HD. The M855 is too, but the penetrator in it usually doesn't disintegrate like the rest of the bullet does in soft tissue, so it might keep going, they do the darndest things. The Hornady 40gr. VMAX is just too light. It basically will punch a tiny hole and then liquify a baseball size or just larger wound under the skin. They say it isn't reliable enough to stop an aggressor. I'd go with the XM193 if I were you, although I use the M855 since it is cheap for me to load, I have a 1-7 twist, and I don't have the problems you have.
Even better for you would probably be a well built soft point in the 55-60gr. range, scooting along at about 3000fps. Shouldn't overpenetrate. A quick double tap will end it fast.
BTW, an M855 round has extreme difficulty, and can often not, penetrate a new cinderblock fully. It can poke a tiny hole in one side, spall the other, come apart in between, and shatter on the other side of the inside of the block without penetrating. Every now and again, one will get though, but it is usually just fragments. Two shots will break down the block though, and the rest will get through. It punches through wood like it isn't there. Bottom line, is to hit your target no matter what you get, because it will go through a regular stick and drywall wall. I've seen a 147gr. all lead 9mm bullet go all the way through a house. One end to the other! I feel the M4 does this best, low chance of overpenetrating target, low chance of missing target.
Just don't get a junk M4. LMT makes a very good rifle for a good price. The Noveske ones can't really be beat. Kind of get what you pay for.