The fix for "over penetration" isn't a caliber or ammo choice; It's awareness, planning, and practice. Over penetration isn't an equipment problem, it's a usage problem. Knowing the possible effects of a miss or a pass-through is part of defensive shooting just as much as it is a part of hunting, if not more so. You have to know your homes layout, where everyone is likely to be, and practice taking firing positions that won't put rounds through people who are not threats.
Practicing firing positions where you might need to shoot through barriers to stop a threat before it can harm your family might not be a bad idea either. Once you've verified a threat and moved your family into a defensive position, if the treat decides to come after you, you need to be prepared for that too.
Once you trained for the conditions, you are free to use the most effective round available to stop the threat, thus minimizing the threats danger to you. If you use a marginal round, you're more likely to be injured or killed by the threat and where does that leave the rest of your family? At the mercy of someone who's just put you down. It also leaves them free to do it to another family.
My preference is a 12 guage pump shot gun with slugs and a weapon mounted light with enough spill that I can point it at the floor and still illuminate an entire room (in this case a Streamlight M3). I choose it because a long gun is harder to lose control of than handgun and generally more precise on target. It's also quick to deploy able to get on target very quickly. Your situation may be different than mine and it might not be viable for you. But that's my thinking on the subject.