If we're just looking at rifle calibers, I'd argue that the list should be as follows:
.22 LR, for plinking and recreational fun, as well as small edibles and Olympic-class match rifles. My choices would be an accurized 10/22 and an Anschutz if some model.
.223, great for plinking, vermin, social work and Highpower stuff. I'd take an AR-15, well, like 3. A flattop varminter, a 16" shorty and my 20" match Service Rifle.
.30-06, because it is enough gun for anything I'll ever hunt, not overwhelming to most shooters, and if you qualify for CMP ammo, not even super-expensive to shoot. I'll take an M-1 Garand, an '03 Springfield and a Ruger 77.
Number four? Eh, probably a .44 Magnum carbine, because I love lever actions chambered to this round. A Marlin 1894PG is perfect for this, as a short range plinker, defense rifle and medium game hunting rifle.
IF I needed a bigger gun, it would be a ,375 H&H instead of the .44. But I don't see that in MY future, so my pick stands!
BTW, if I can only have four cartridges, I'm not gonna be stuck with only four rifles!