Hi. My M1 has been fed nothing but my handloads(using IMR4064) almost since I got it 30 some years ago. Other than the factory ammo I used to gather brass. 220 grain Silvertips included with no damage to the rifle. (What one of those does to a ground hog is amazing. Poorly sighted, one shot took off the bottom of the head. The second shot blew the offside, gone.)
Up here, there has never been any case lots of M2 ball(worked in a gun shop that had case lots of .303 and 7.62 ball. Never any .30-06 ball of any kind.). Factory hunting ammo was used by everybody with no fuss. I've never seen nor heard of an M1 being damaged by the ammo alone until I got on the Internet forums years ago.
In any case, no milsurp ammo is made for great accuracy. It's made to be reliable and within accuracy specs. Personally, I wouldn't want it for that reason.
"...my buddy's '03..." 168 grain match bullets with IMR 4064 give one hole groups out of my '03A4. Doesn't have an issue scope though. Has a Bushnell Scopechief 2.5 to 8 on it. The rifle came with the issue base, but no scope or rings for $175Cdn., in the early 80's. It's so sad that my toys are my retirement fund.