Just Three
Well, there's really no accompanying scenario to impose limits, but I'm gonna assume there's a reason you're stuck with only three.
At the low end, I'm torn. The .22 LR is a nice general plinking and practice round, fine for pest control and stuff. Still, you might want a little more beef, so the .22 WMR is in the mix there. So, going with what rifles I actually have, I guess we'll take the .22 LR for now, .22 WMR perhaps when I become better acquainted with it.
In the middle, for deer and such, I'm gonna break the rules and take a .357 carbine. Fired from a rifle, the .357 has rifle-like capabilities out to 150 yards, including deer. This gives me a lower powered load (.38 sp) I can use for medium varmints and bigger "small" game.
If the referee threatens to throw me out of the game for that, then my fallback is the .30 carbine.
Now, for large, although I have a healthy respect for the .308 and .30-06, I have neither. Sticking with what's in the safe, I'd use the .30-30 for the big stuff. I will acknowledge that it's not the hitter that the bigger cartridges are but, over time, the .30-30 has taken several of just about everything. There is also the little matter of I have to shoot the thing. I can shoot a .30-30 and handle the recoil. Having never fired the harder hitters, I don't know where'd I'd stand on tolerance for that.
So, for now, until I have broader experience (and a bigger safe) . . .
Small: .22 LR
Medium: .357 carbine
Large: .30-30
And, for what it's worth, I have all of those in lever rifles.