There's a legitimate physiological explanation for this. There is a muscle attached to the stapes bone that pounds on your eardrum (stapedius muscle). If you are anticipating a loud noise, the muscle tightens and limits the movement of the stapes, reducing its impact on your eardrum. When you're expecting a loud noise (i.e., when you're shooting), your body unconsciously reacts to reduce the impact on your ear. When someone else is shooting you don't know it's coming, so you get the full effect.
