DEFINITELY GO WITH THE '06.
I've got a lot of rifles, but when it came time to go to Wyoming earlier in late Sept., I took two rifles- my Remington M7 in 7mm08, and my Colt Lt. rifle in .30/06. My favorite of the two is the little Remington, but the '06 is the better elk gun of the two. Just a personal thing.... stock fit, trigger, ect....
The recoil from a light '06 though noticeable from the bench, in the field dosen't amount to much. My Colt weighs 6.5lbs with Burris 2.5-8 scope, sling and 5rds of ammo. Recoil is present, but not really remarkable with my chosen elk load with a 180gr bullet over 55.0gr of IMR4350, for 2,800fps mv (measured at 20' with chronograph). Sighted +3" at 100yds, it shoots sufficiently flat for your intended purpose (very much like the elk hunting I've done....).
I've got other rifles, such as a 10lb MkX Mauser in .338/06, and a Savage M110 in .300RemUltMag w/26"bbl. You carry the rifle MUCH more than you shoot it, and the weight is cumulative over a week or more of walking 10-13mi a day at 7-10,000' elevation up and down hill with gun, pack, ect......
My .300RUM recoils so much that from field positions shooting at game, it's drawn blood on both ends every time I've used it to take game, so it dosen't go afield much....(scope objective is "cushioned", but it tears the skin instead of cutting....) Just to the range so that mine really IS bigger than "yours", if you know what I mean.... It's a great rifle, shoots flat, (no flatter than my .257wbymag though), but it does recoil more than a similar weight .458winmag. I know, I've shot both......
The '06 offers a bit more punch with the 180-200gr bullets, and I've owned and shot more game with it than the .308, so I'm more "confident" with it.
For a good inexpensive yet accurate and lightweight rifle, I'd get the Stevens M200 in '06 and put a set of Leupold/Weaver aluminum bases and rings, and either a Nikon ProStaff 3x-9x or a Leupold VX-I scope and call it good. Also, I suggest the Federal 165 or 180gr Nosler Partition "Premium" ammo.....if you don't reload for hunting, and whatevers cheapest for deer and what-ever.... as long as it's the same weight bullet. (I like the Remington CorLokts..... but the Federals and Winchester promo grade ammo are equally as good.)