One other thing to remember with a scope that most people don't realize, concentrate on the center of the crosshairs as the shot goes off. Just like with open sights you need to pay attention to the natural point of aim (where the rifle's sights lay naturally) and the sight alignment (how the front sights align with each other). With a scope you don't have a rear and front sight that can be aligned with each other so you have to concentrate on where the center of the scope is pointed and you do that by making sure that the center of the cross hairs are on the point of aim. This becomes habit pretty quickly once you're aware of the need to do it but for most people it only makes a small change in the group size. While the improvement will be small, it will be an improvement and any improvement puts a smile on your face.
By the way, people get in the discussion about group sizes and flyers and how many shots are needed to build a good representation of what the rifle will do but I believe in keeping it simple, 5 shots is a good working group size for the average hobby shooter. 5 shots is a good compromise that keeps the cost of ammunition down and still provides a good representation of the groups that your rifle will do. As has been pointed out by others though, the more shots you take the better idea you get of what the rifle, ammo, and shooter can do. Statistically we normally use what is called a small sample of the whole, unfortunately the whole, for a statistician, is hundreds of rounds and it's pretty unrealistic to shoot that many rounds each time we test a load for accuracy. This is where people come up with this comment that flyers don't exist, I agree that it takes a little experience to determine what's a flyer and what isn't and most of what people refer to as flyers aren't, they are simply shooter error but flyers do exist. Sometimes a rifle will throw a shot arbitrarily and so it's important to recognize when that happens and when it's just the shooter's lack of concentration.
The hard part here is the OP isn't sure if he's getting flyers and honestly, neither is anybody that's isn't the shooter, only a shooter can make that call. And that's actually the solution, making the call. A lot of people will mention that a shooter should be able to predict where the bullet will impact (making the call) and they should, you should be shooting slow enough to say with absolute certainty that your shot will go where you intended. If you pay attention to your position, the natural point of aim, the sight alignment, the trigger squeeze, the shoulder pressure, the cheek weld, your breathing, the fact that your muscles are relaxed, the position of your feet, your shoulder alignment to the target, and all the other things that are necessary for consistent shots, then you'll be able to call your shots every time. And when your bullet impact doesn't go where you called it, then it's a flyer. It takes time and lots of practice to call your shots well and in the beginning you will find that your calls wont match the impacts very well, this just helps you develop your checklist of things that you need to pay attention to every time you shoot.
All this long winded stuff is what you need to make perfect target shots, it's not what a hunter does or what a plinker does. Unfortunately even a plinker or a hunter should use this technique when he's developing a load and he also needs to use larger groups to determine when he has a the kind of group issue that you have.
In my opinion you either have something moving on the rifle or you've got a problem with your concentration and each shot isn't exactly like the last.
