I recommend 30-06 or .308. You can always find deals on loaded ammo or brass for either since they are military surplus rounds.
If you are looking into the fine art of reloading and benchrest shooting, I'd say jump right past any standard cartridges, do some research and buy a few guns in one of the calibers benchrest shooters shoot. It looks like 6.5X55 is in that camp, but I am not of that world, so keep researching for that route.
My $0.02 - 30-06 is the way to go. My first grown-up rifle was a .270 (after a .22 and a .244). I use it for deer hunting with a sight-in once in a while, but I burn through about a million% more 30-06.
.308 is available on a few more platforms than 30-06, so it's worth exploring. But you can buy a Garand in 30-06 (
www.odcmp.com). You have to retrofit it for .308.
A "one gun" plan? M1A in .308. Deer, plinking, competition, self defense.
There are not a lot of semi-automatic platforms or shooting competitions for .243 or .270 if that matters.
Edited to add: To answer your original question - .270.
.243 is good for starting shooters or if you want just ONE rifle for both varmint and deer hunting.
BUT! I see no need to have just ONE rifle to serve both needs when you can buy a NEF Handi Rifle for $200, a Stevens or a Mossberg for less than $300 or a great rifle from Remington or Savage for less than $500. Even cheaper when used.
Even on a tight budget, you can move the scope between rifles and always have a nice backup for either a deer or varmint hunting weekend. A disabled gun without a backup can ruin a spendy hunting weekend.