I have only been shooting twice without hearing protection. First time was five shots from my dad's .243, and I was in pain--this was at age 18. Second time was about a year later, I'd forgotten them and was shooting an AK and Mosin. The pain returned, but not as bad.
The real eye-opener was a Slayer concert about three years ago. I had noticeable hearing loss for about three days afterward, and my left ear still isn't what it used to be. I'm 26.
I hate muffs, though, as they're hot, uncomfortable, and whenever I use them for rifle shooting, I have to choose between either a good cheek weld or sound dampening--never both. Must just be the shape of my head or something.
Instead, I use plugs exclusively, specifically banded ear plugs, for great noise reduction and quick on/off. Got my first pair when I worked as an aircraft refueler, and they work so well your head can be two feet from a running jet engine and it won't bother you at all (except, of course, that your head is two feet from a jet engine).