I've used and loaded for the .30/06 since 1975. I prefer IMR4350 with 165-180gr bullets. With the 150's and lighter, it's either IMR4064 or Reloader15. Neither my brother who also shoots an Garand in HighPower nor myself have gotten as good of accuracy from H4350 as the IMR4350. (Don't use the 4350's in the Garand....)
For accuracy, my Savage M110 LH, prefers 50.0gr of IMR4064. With a 165gr bullet, it's 56.5gr and with 180gr it's 54.0gr of IMR4350.
These are well below max, and less than what I run through other '06's.
The Savage lacks the camming action with the bolt opening of other action designs and my gun has a very short throat, so it won't tolerate loads that other guns will. Some factory ammo is too hot for my gun... Not a slam against the Savages, I have several, just an observation. My .243 was the same, as well as my .300RUM.....
Do start with the recommended starting loads, and don't try to "hot-rod" your rounds. With the '06, it isn't even needed...
I've not gotten as good of accuracy with either IMR3031 or H or IMR4895 or Varget as with IMR4064. Same goes for the .308 as well. Your results may be different however as many still use the H4895 in the '06 and .308.
Magnum primers aren't needed either, just use the standard large rifle. I prefer Federal, and Winchester, it that order.... Not crazy about the CCI's but I have used them with the others weren't available.
Most brass is about the same, however, Mil-spec brass will need the charges dropped about 1.5-2gr below commercial brass due to incresed thickness in the base area.