Imagine you were going to create the perfect firearm for home defense. What would you want? You'd want it to be reliable. Because click instead of bang is not an option when it is your family on the line. It should be shoulder fired because they are so much easier to be accurate with under duress. But for use indoors you'd want it lightweight and compact. It should be easy to shoot, easy to maintain, and easy to adjust and modify for different users. That way your spouse can use it too. Red dots and reflex sights are much easier to use and quicker to acquire than iron sights, and you might want to add a white light as well. It should be accurate, because you're going to have to own every round you fire. But it should still have decent capacity, because no one ever got in a gun fight and wished they had less ammo. And what about the cartridge? We need something powerful enough to take the fight out of someone quickly, but not so powerful as to be irresponsible in an urban setting, or so powerful as to produce excessive amounts of recoil and blast. Congratulations, you just invented the AR-15 in 5.56.
The AR-15, and other lightweight semi-auto 5.56mm carbines have some advantages over both handguns and shotguns that bear consideration. A carbine can be found significantly lighter and shorter than a shotgun. Even in pistol format, a braced pistol in 5.56 can be had shorter and lighter than something like a Mossberg Shockwave. The carbine isn't quite as terminally effective as the shotgun, but has much better capacity and lower recoil. It is also typically easier to add optics too. The AR will require less from the user to be reliable than a pump action shotgun, as long as it is maintained in advance. The ease of use combined with a more nimble and responsive package in a low recoil cartridge means that the AR is just easier to use in tight spaces, easier for new people to get accustomed to, and easier to acquire and maintain proficiency with. The 5.56 cartridge is well suited for urban environments. It is a small caliber round with projectiles that typically have thin jackets. When driven at the velocities the 5.56 is capable of pushing them, the projectiles typically deform readily, either fragmenting and losing mass or expanding and increasing surface area and drag. Either way this causes them to lose momentum quickly. As a result, many rounds for the 5.56 penetrate equal to or less than many popular defensive handgun rounds. Handgun rounds tend to be larger in caliber and heavier in mass. They lack the velocity to do the cavitation damage the centerfire rifle is capable of and the energy to be reliably effective. But they have a decent amount of mass, which means they maintain momentum more readily, and tend to penetrate through many common household barriers surprisingly well. The 5.56 will be much more effective terminally than pistol calibers, even when fired from carbine length barrels, because the 5.56 produces enough velocity to damage tissue through hydrostatic shock, and deposits two or three times as much energy two or three times as fast as the pistol rounds.
So the carbine is more effective than the pistol rounds, by a wide margin, and more handy indoors and easier to use than the shotgun, again by a wide margin. And with proper ammo, like a Black Hills Mk 262 with the 77 gr SMK or the Federal FBIT3 round with the Trophy Bonded JSP, the 5.56 will also be safer indoors than most handgun or buckshot rounds. That is why the carbine has almost completely replaced the SMG and the shotgun for police use, and why over 5 million Americans own over 15 million AR-15s.