It's hard to speculate and give sound advice beyond what's already been given without actually being there. Groups could lean left or right based on trigger finger placement as well, for example.
Keep the same distance, since you can hit the target in the vicinity of your aiming point without any real problem. Then plunk your keister down on a seat and shoot from a bench rest, supporting your arms to minimize arm movement.
Then, with a PROPER grip and PROPER trigger placement, concentrate on maintaining your sight picture through each and every trigger pull. Take your time while you do this and if you find yourself getting tired, take a break.
Your groups should tighten up when you do this.
Once you've gotten yourself into shooting with tighter groups this way, make it a point to dedicate a portion of your target shooting to shooting this way. The purpose of this is to develop your hand and trigger control, along with your concentration, such that you are able to go through the motions of acquiring your target sight and holding it throughout your trigger pull. Shooting supported from a bench allows you to isolate other body movements which may interfere with and confuse you as you develop this control.
Once you've established a baseline by bench shooting, then shooting standing up will be both easier AND it will allow you to more easily figure out why your hits are scoring the way they are.