Sounds like you're anticipating the recoil to me, and maybe pressing the trigger with the 'wrong' part of the finger.
The wrong part of the finger would be described as the point on your finger where you end up applying non-direct (ie, lateral) pressure on the trigger. Depending on the pistol and how you're built, it's going to vary from pistol to pistol due to pistol size/fit.
As for anticipating recoil, I'm going to guess you're doing one of two things, actually:
1) You're "peering over" the gun to look at the target after the shot, and as such, push the pistol down to get a look-see.
2) You're "pushing" the gun in anticipation of the recoil, and as such, the muzzle gets pushed/falls down.
Also another option, I imagine: if it's a polymer pistol, the increasing forward weight of the gun as you unload the pistol might lead to a consistent grip result in a shifting point of impact.
Just a couple possibilities.