I'm going to suggest that it's the bullets.
My Remington M700 ADL synthetic is an absolute tack driver with Sierra bullets up through 55gr including the Boat Tail soft point and Boat Tail Hollow Point bullets. Groups will run 0.4" or so for 5-shots with a 50 or 55gr bullet such as the Sierra or Hornady flat-based bullets w/o a cannulure.
However, it will not hit the target at 100yds with a 60gr Hornady V-max, and a 55gr V-max will shoot 3-5" groups. Same powder charges and similar seating depths.
Rifle has a 1/14" twist barrel which is common to most .22-250's.
I have a 1/12" twist Remington M7 in .223 that will shoot at best 1.0" groups for 5-shots, but it will shoot 1.0-1.3" groups with the 60gr V-max bullets and 1.0" groups with any decent 55gr bullet including the Hornady 55gr V-max.
My 16.5" bbl AR15 with a 1/9" bbl will shoot any decent bullet under 70gr into very, very small groups. But "scatters" 77gr A-max bullets at 100yds.
Hence, I suggest you use at maximum, a 50gr Poly-Tip bullet through a 1/14" twist (.22-250 or some .22-Hornet's) barrel.
Or, get a 1/9 or 1/8 twist barrel for your T/C frame.I noticed that you said it was the last 5 rounds that did this. Combine a marginally stabilized bullet and a fouled bore, you'll get keyholing.
Also, I had a friend that 4yrs ago got excited on a prararie dog town with his Savage M112 HB in .22-250. He went through about 400rds of 40gr V-max bullet over 40.0gr of Varget. When he got home and cleaned his rifle, he went to the range and tried to reset the zero and work up a load for the 50gr V-max. He couldn't get it to hit the target with anything. He later found out that he had shot out the first 6" of the throat of his barrel. He had to sent it back to Savage and had them rebarrel it.
He had a total of about 1,000rds through his barrel!!!