Are you shooting off-hand, or Weaver stance from 10 yards?
If so, in my opinion, a 4" is not bad for self defense.
Though you shoot a lot, and shoot other guns better, I'm sure you know that each gun is different.
Put another 500 rounds down range and see what happens.
In my experience, small guns are much more difficult to learn to shoot accurately.
I've just passed 600 rounds in my PM9, and my accuracy seems to get better each time I go to the range.
My daughter hasn't shot it much, but from 6 yards, she easily got a rapid fire group of 4-6". She can do that with her snub nose Taurus also. I was very happy with that. I have no doubt that she can defend herself with a handgun, if the need arises.
If you are getting 4" from a bench, that is not so hot.
I have heard of bench rest groups fo 2-3" with the PM9, but I'm not that good of a shot myself.
I didn't get the PM9 for a target gun, or a tack driver.
I do want a concealable self defense gun that I can get accurate hits with, at reasonable self defense distances.
Currently, if I work at it, I can get get all 7 shots to touch from 6 yards, using cheap Winchester 115gr. ammo, if I'm having a good day.
Just for kicks, I tried some Speer +P 124gr. Gold Dots from 25 yards, Weaver stance, slow fire, at a B27r silhouette.
My gun seems to love that bullet.
I had a 6" group in the middle of the target!
If I fire rapidly from 6 yards, and get a 4 MOG, (minute of badguy), I'm happy with that.
My PM9 is the most accurate tiny gun that I have owned or fired.
(Compared to a NAA Guardian, NAA .22 revolver, KelTec P32, S&W9-something, S&W Centennial, and Taurus 94 Ultralite.)
My Glock 26 may have been more accurate, but it was a much larger gun.
I'll be the first to admit that I'm not a world class shooter.
If I was, I suspect my PM9 would turn in even better numbers.