When you say "all size game", you're talking anything from elephant in Africa to mice in your garage. That's a heck of a size progression!
From small to large, I'd say you'll need:
1. .22 LR: for the mice and small varmints at close range.
2. .223 or .22-250: for varminting at medium to long range.
3. .308 or .30-'06: for all small to medium deer, antelope, etc. You can go up to elk-size animals with ease with these cartridges, and caribou and moose are not out of the question.
4. A heavy rifle for the really big stuff. My personal choice would be the .375 H&H Magnum, but some would prefer a stomper such as .458 Lott or .470 Nitro Express. Personally, I'm confident that the .375 will do the job, if I do my part and place my shots where they need to go.
5. 12ga. shotgun, for all bird hunting needs.
That would be my minimum "all-round" hunting battery. I'd add a decent revolver, probably in .44 Magnum, for close-range defence and other purposes while out in the field.
A good case can be made for cartridges in between these "stops" on the cartridge continuum, for specialized use against particular game or in particular hunting conditions. For example, a good .30-30 is a thing of beauty in woods hunting, and a .45-70 carbine is wonderful for hog hunting in close quarters. However, the above battery would get the job done if necessary.