Not quite true. The lighter bullets will require a greater charge of powder to achieve the same power factor as the heavier bullets. This powder has an influence on recoil, too, beyond just its contribution to the momentum of the bullet. The gasses generated by the burning powder have mass. They also have velocity - a lot of velocity! More than the bullet itself. So they punch above their weight (almost literally) in terms of their contribution to recoil. Thus, when you include the jet of gas coming out of the muzzle, the lighter bullet using a larger charge of the same powder than the heavier bullet will have more total, actual recoil... separate and apart from the subjective perception of more "recoil" coming from the blast and flash.
Here's an interesting article that tested this concept mechanically.
http://www.shootingtimes.com/reloading/power-factor-recoil-bullet-weight-gives-edge/
Also, note that all of the above gets turned on its head when a compensator or ports (any device that
uses the gasses to mitigate recoil) comes into the picture.
ETA: There are also some problems with the duration/rate of acceleration explanation for why shooters would experience a difference in recoil between two identical power factor loads.
Especially in a semi-automatic gun, where the bullet is gone from the barrel before the vast majority of recoil forces are imparted to the shooter (which doesn't happen until the slide hits the rearmost end of its travel).