My personal answer that works FOR ME would be as follows:
- Shotgun
- .22 rifle
- Centerfire rifle
- Handgun
For a good little while I was working with just the above. I had a .38 snub, a Marlin 336 in .30-30, a Marlin 60 and an old used Wingmaster 12 gauge. For a core battery, I could have easily just stopped right there (and yes, .357 is more versatile and preferable I agree).
You can debate caliber for your centerfire rifle. In the Southeast, I believe you can get by fine with the .30-30 although of course a more powerful caliber doesn't hurt.
I think it's hard to understate the low cost and portability of .22LR. Yes, you can do a lot of the same things with a shotgun, but try carrying around 500 shells compared to your average brick of the lowly .22; there is no comparison.
Also the portability of a handgun is what it brings to the table. If you're trying to be discreet at all, most any long gun is not going to cut it. A handgun is much more likely to be there when you need it. Not to mention it is quite a bit easier to carry a long gun + a hand gun than it is to carry two (or more) long guns. I guess if you are going to be 100% in the woods and totally living off the land maybe you can forego the handgun. YMMV as always.