K-hornets experience most closely matches my experience.
To get the accuracy you are looking for try this:
Size your brass to just touch shoulder of the case.
Use Federal 210 or 210M (match)
Use the Nosler Balistic Tip bullet, or Sierra 150gr FLAT BASE.
Seat to at least .310" bullet inside the case neck, Usually 3.250"oal with the '06 and 150gr bullets. The deeper seating has given me better accuracy with most .30/06's as they tend to have long necks. I've had similar experiences with .22 Hornet, 7mm-08, .30/06, and .338-06. With long loose chambers, you get better accuracy with the bullet guided into the lands by the case neck holding it "centered" than with seating it to touch the grooves.
Use 52.0gr of IMR4064. This has been most accurate load in at least 10 different .30/06 rifles I've worked with. Max is 53.2-54.0gr depending on the source of data. You'll do better working close to max with the '06 as the data is milder than for other more recently released cartridges. Ditto for the .257 Roberts.
The 165gr most accurate load has been 56.5gr of IMR4350. (57.5-58.0gr is max), so, with the 165's, you just hadn't gotten there yet.
Good luck with your efforts.
Just don't forget to use some of them to hunt with. The deer/elk don't know the difference between 150 and 165gr bullets, nor 1" vs 2" accuracy, nor 2,650fps vs 2,900fps !!!!
BTDT !!