I agree with the flinch diagnosis. It's probably the most common problem among infrequent to casual shooters. As a slice is to golf, a flinch is to shooting.
I'm of the view that flinching starts with the eyes. You have a very normal human physiological reaction to close your eyes when there's a VERY loud noise and a sudden jump of something in your hand. You may have overcome it with a 9mm in a mid-size gun, but a smaller gun in a louder/snappier cartridge brings it back.
As with any flinch, what you mainly need to do is gradually train yourself to be OK with the gun going off. The eyes are, again, the key. Next time you shoot that gun, don't try to hit anything in particular (other than a safe backstop). Just try to see the gun go off. Try to see the muzzle flash. Try to see the brass eject. Try to see the front sight rise in recoil. All of those things should be perfectly visible if your eyes are open. If you just can't do it, go back to the 9mm for a while to rebuild some "trust" in your brain - or, better yet, a .22.
Some people can overcome a flinch fast. Others (like myself) require many, many rounds and several sessions. It may even come back on you from time to time, especially if you go a long time without shooting.
Once you can keep your eyes open through the shot, chances are you won't push the gun off-target. And if you do, you'll see it so immediately that your brain will really understand it, will "believe" that it's pushing the gun, and will quit it pretty quickly. You'll be able to aim through the shot, rather than getting set, and then closing your eyes and jerking the trigger quickly so the shot can be over and you can re-open your eyes.
Go watch the gun go off. Find out how much you can see.