Most recoil is felt between the ears. If you believe something is going to kick hard, it will. A 30-06 of reasonable weight, 7.5lbs -8 lbs including glass and mounts shouldn't be bad at all. Go much lighter and it could be an issue.
The new high-tech recoil pads are great. Stock shape is important, most modern guns have very good stocks and are much better than older designs. I've owned and shot 6 lb 300 win mags with good pads and modern stock designs that were very tolerable. My 5 lb Kimber 308 is not bad at all and will recoil more than a standard weight 30-06.
The X-bolt weighs 6 3/4 lbs without optics. Add about 1-1 1/4 lbs for glass and mounts and you are at 7 3/4-8 lbs. Recoil should be a very reasonable 19-20 ft lbs. Just for comparison that is considerably less than the 30 ft lbs or so generated by 12 or 20 guage buckshot. About the same as most bird shotshells. A 308, which most people find mild recoiling is only 1-2 ft lbs less.