Another vote for barrel-twist and bullet-weight compatability issues. If you have a 1-9" twist, drop back to the 75gr and under bullets and you will be fine.
I toss 80gr Noslers out of a 1-7" twist AR-15 with a 20" barrel to shoot at 600 yards. For 200 and 300 yards, I have gone back to using 69gr Noslers over 25gr of Varget. These chronographed 2840-fps average for me, and while I think I could squeeze a few more FPS out of them, they shoot well there.
And quite honestly, I've come to the opinion that Varget and RL-15 are practically interchangeable in .223 with heavy-bullet loads. Whatever you can do with one, you can do with the other. Just another option to try if you're loading out of 1-lb canisters and want to try something different.
Oh, and what OAL were you loading at to have them "well-compressed" at 23.5gr? The 80s at 2.55" are far from compressed for me with that powder charge. At 24.5, you should start to see compression.