I see shotguns as having one main advantage over carbines, power. The effects of a slug on target are most impressive. A second advantage is price. I can get a shotgun I would feel good about using defensively for a couple hundred dollars. For an AR I'd be looking at around $800. For an AK probably close to $500. I own shotguns, like shotguns, and would feel well armed for just about any defensive situation I'm realistically going to find myself in.
That said, I have been swayed more and more to using a carbine if I have the choice. I see a proper carbine as having the following advantages:\
1. weight: My defensive shotguns weigh more than my defensive carbines. I feel like I can use either but the lighter gun is handier and faster. Importantly, my 105 lbs girlfriend can much more easily wield a nice light AR.
2. Size: related to the above a carbine is typically going to be smaller which again makes it a faster handling weapon. You can get an SBS but unless you are using a box mag fed gun one starts losing capacity as well. Further if an SBS is an option so is an SBR.
3. Manual of arms: I think I carbine is easier to really run well than the majority of shotguns.
4. Over penetration: A carbine with the proper load poses less risk than any load I would use in a shotgun for defensive purposes.
5. Ammo capacity: shotguns are low capacity weapons. One is probably looking at 7-8 rounds in the gun. That may well be enough. I don't get to choose my gun fight though and I am not going to make assumptions that I can not be certain are true. For that reason, within reason, more ammo is better IMHO. A carbine can hold 20-30+. Also a spare mag means another 20-30+. A tube gun might be able to carry another 8-10 rounds on it.
6. Ability to suppress: The only effective shotgun suppressors I've seen have resulted in pretty large weapons. One can put a can on a AR and be below or near the 18.5" barrel a shotgun has. A suppressor has a serious benefits on a defensive weapon.
7. Less recoil. A carbine is faster to make followup shots and transition from one target to the next. Again it is also more manageable for many than a shotgun shooting slugs or buckshot.
8. Ability to penetrate soft armor. This may never be needed, but again I don't get to pick my gun fight.
A rifle has more range but that is not really a concern for almost anything that will be seen as a justifiable shooting.
A rifle is more precise but this shouldn't be much of a concern at realistic self defense distances.
Also one could largely negate the power advantage of a shotgun by picking a powerful carbine that would otherwise largely retain the other advantages of a carbine. Say a .458 socom AR or the like.
In sum, I have both. I've trained with both. I would feel comfortable with either but if I had my druthers it would be a suppressed carbine.
Shotgun. While I DO like carbines, I'd rather not take out one of the neighbor's kids due to an over penetrating .223 exiting a bad guy's cranium. It's extremely bad karma.
Carbines with the right loads over penetrate less than shotguns with viable defensive loads.
Although a slightly different context I'll just add that none of the swat guys I know use shotguns for anything other than breaching doors (well actually one told me its his choice for shooting dogs). They are using MP5s, G36s, and various AR/M4 carbines.