this is a complicated issue; i read an in depth story on first shots, from a huge almalgam of weapons, about a year ago. clean, not clean, not clean for 15 minutes, not clean for an hour. not clean for a week. semi, bolt, pump, lever, rolling block, you name it. Now then, the conclusion was this, after making sure that all screws tightened, all scopes perfect, all action screws torqued at about 25 lbs, in other words, all other things being equal, the best shots mostly came, from a clean, cold bore, rested at least 1 day. and when I say clean, i mean , inspected with a light and a bore scope, clear of copper. When I say mostly, it is that, some weapons, I think a Ruger, and something else, just did not improve, and they scattered groups. and then again, another ruger, and a couple of other weapons, threw the first shot several inches, like yours, and then grouped down, but at least the first shot, was allways the same, and very predictable. The rest of the weapons, were just closer to first shot poa, and then with the group. Some weapons were dead on , on the first shot, and they were not big cals, nor even mediums, it was 243, 223, and some weapons, between 6 and 7.5 cals, not sure a 308 even made it into this group. and then a few big cals made it, like 375's, and I think something like a 45/70 and of course these were regular, off the rack weapons, not specials like trg's, or AAI's, or anything like that.
In other words, all weapons liked to be very clean, and rested before the first cold shot, generally speaking, but it did not make any of them much better or perfect, they either were allready pretty close, or not. And a few did not improve at all.
So my advice to you is , check all screws, check torque of takedown screws, thorougly clean the weapon, especially the bore. 1 more thing you could do, is to take some aluminum tape, make some bandaid sized strips, and put in around the front,back,sides,and bottom of the action, about three strips thick, like stacking bacon. ....mmmm... bacon...
and then put the rifle back in the stock. I would now go out and shoot the crap out of it, at least 50 rounds. Now take back to house, and without taking it out of the stock, thoroughly clean the bbl., again. we want the rifle to be comfortable, and undisturbed, in it's new bedded with aluminum tape, stock. again, check all the screws, and the torque of the take down screws.
So we cleaned it all real good, made the action/receiver area of the stock, much tighter and snugger with the tape.
I did not mention it before, but you better have made darn sure, that the bbl is free floated, period.
So after it is all again, cleaned, resnugged in to the stock, and all screws and torque checked, wait at least 1 day, then shoot the cold shot. If after all this, it still does not change much, then you have what you have. If it gets closer by half or better, then you could spend some money, and have some things done to it; have the chamber professionally redone, have the bolt squared and trued, have the muzzle redone, have the stock properly bedded.