there are 3 types of 12 gauge shells: 2 3/4 in, 3in, 3 1/2in This is the size, in inches, of the hull after it has been fired and opened up, so while a 3 inch will fit unfired in a 2 3/4th chamber, it will be BAD.
Read the barrel, or action, it will be stamped somewhere with 12 GA 3IN or something.
If you cannot find it, use 2 3/4ths ONLY.
Now, it is a good idea to use 2 3/4ths anyways, because you can squeeze one more shell in the gun that way.
As others have said, there are some special purpose shotguns that have rifled barrels, while it is unlikely yours is one, take a look down the barrel and see if you see riflings. Rifled shotguns = slugs only. Smooth = slugs or shot.
Slugs are one big solid projectile, hits hard, punches through stuff, but moves slow so it isn't a whole lot of good beyond 100 yards.
Shot is a bunch of small projectiles. A very standard loading will launch 1 1/8th OZ of lead. Sometimes they go with 1oz for a bit faster speed, sometimes tehy go with 1 1/4th oz for more pellets, but whatever.
Here's the deal, lead is lead, oz is oz, so for a given 'payload' of 1 1/8 oz of lead shot, what is going on, what is the difference?
Shot size is the difference. the smallest is 'standard shot' which is given a number value. 10 is the smallest all the way up to 1, with some b, bb, t, tt etc to denote stuff a bit larget than 1 After that there is another scale called buck shot, again big number denotes smallest size, from 4 to 0, then 00, and 000 to denote larger yet.
So with a given 1 1/8 oz of lead shot, if you have a very small shot size, you have many many many projectiles. If you have very big shot size, you ahve a lot less projectiles.
So more is always better right? NO! You are still throwing the exact same weight of lead. If you placed the barrel inches from the target, it would pretty much all be the same. However, after a few feet the shot is no longer all in one blob, it starts to spread out. The smaller the shot, the GREATER total surface area the entire load has exposed to air. Smaller shot looses it's velocity faster, and each individual pellet often doesn't do much. Of course, collectively, many small pellets can do a lot...as long as the target is close ehough that they aren't slowed down too much by air resistance. A slug at 150 yard shots will probably miss the target, but what it does hit it will hit hard, same with a handful of large buck shot, but very small shot at 150 yards...well if it gets there it won't do much. Look at Dick Cheney's hunting buddy.
So for home defense, it is a balancing act. Just like there is more surface for the air to act on with small shot size, small shot is also looses more when it needs to punch through a wall. This is normally a good thing because you DONT want to punch through a wall...but then you are also loosing the ability to punch deep into a human body and do damage enough to stop the guy...especially at longer ranges.
Of course, in a house, what kind of ranges are you talking about? Short.
I've chosen #4 buck as my comprimise, I know guys who go with #1 standard as their comprimise, but whatever works. #8 standards, while real small, at close range (bedroom back wall to bedroom door) are capable of taking off the top of a human skull. Unpleasant as it is take a look at photos of hunting accidents, these are most often small shot (they were planning on shooting birds after all) at very close range....not watching where the muzzle was going when the hunting party is getting in the truck, or over the fence, or in the boat, etc.
The next step is to determine what you want to use a given load for.