So you are claiming you can "feel" the difference in something where "fast" and "slow" differ by a few tens of microseconds?
Here is a recoil calculator:
http://www.handloads.com/calc/recoil.asp
Generally for handguns the powder charge has a fairly small impact simply because the charge mass is small compared to the bullet mass, but the escaping gases have much higher velocity so the effect is not totally negligible.
Plugging in the numbers for a 230gr bullet at 850 fps with two mythical powders that produce the same velocity with charges of 5.0gr and 7.8 gr for the other. The 5.0 grain load has a recoil impulse of 0.96 lb-sec while the 7.8 gr load is 1.01 lb-sec
Doing the same for 124 gr bullet at 1100 fps and powder charges of 5gr and 10gr the recoil impulses are 0.69 lb-sec for the 5gr and 0.78 lb-sec for the 10gr load.
I'm of the opinion than most of the perceived recoil differences other than these maybe 5-10% changes in recoil impulse is mostly your reaction to the muzzle blast and flash from the "uncorking" pressure.
The recoil impulse is the "thrust" imparted to the firearm from the shot. Its independent of the weight of the gun and how you grip it.