OFFGRID, I sympathize with your frustration trying to find a handload that matches or beats a factory load that your particular rifle likes. I, too, went down that road decades ago, also with a 30-06 that seemed very picky about what ammo it liked.
I finally found an accurate load, 58.0 gr of H-4831 with a 165 gr Nosler Solid Base spitzer with a CCI-200 primer in Winchester brass. I killed my first buck with it and was thrilled. Until I gained access to a chronograph and found out that my accurate load clocked only 2400 fps out of the 20” barrel of the Model 70 Winchester lightweight carbine. What? According to the manuals, that load should have been at least 2700 fps, if not 2800! (Never mind that the deer died with one shot behind the front leg and passed through the offside shoulder). Factory ammo had been less than impressive, accuracy wise, and eventually I figured out that my shorter barrel was not going to provide the velocity that a 24” barrel would in handloads.
I picked up a box of factory PMC 150 gr soft points, and was pleasantly surprised by the accuracy they got. So I pulled a bullet out of one of them and dumped out the powder. The charge weighed only 49.5 gr and was a short grain extruded type. Perusing various load manuals, I decided to try a similar charge of IMR-4895 with a 150 gr Nosler Solid Base bullet and surprise! Decent accuracy! Velocity was still lower than I would have liked, but it was consistently accurate. Later, I hit on an even better load, using Accurate 2700 that also gained almost 200 fps over the PMC factory 150’s and my similar handload. Bottom line is, if you find a load that your rifle likes, and you are satisfied, go with it. Or, if you want to keep trying to improve on it, join the club. Send me a pm if you would like to see my load data that I experimented with over the 12 years that I played with my 30-06 (from 1985 to 1997). You may find it interesting. Some of the powders and bullets are no longer available, but a lot of them still are.
My last piece of advice is, if you do opt for factory ammo, buy a bunch of the same lot no. If you buy the same ammo 3 years later, it might be loaded with a completely different powder and give very different results in your rifle.