I say, the more the merrier. However, I don't keep very much on hand for calibers I don't shoot often. I probably only have around 1000 rounds for .22LR and maybe 250-700 rounds each for my "rainy day" calibers (5.56, 7.62x39, 9mm, 12ga).
For some reason, I can never pull together enough money...