Those of you with hearing damage shouldn't beat yourselves up too much.
Most of my shooting without hearing protection was with a single shot 12GA during hunting trips. I was between 12 and 15 years old for most of those trips, probably 15 trips total. The shooting would make my ears ring a bit, but I don't know how much permanent damage ever occurred.
Then when I was 18, on the night of my high school graduation party, I developed an incredibly painful ear infection which made me temporarily deaf. This infection took four rounds of different antibiotic treatments over a six weeks to eliminate. What's more, I injured my ankle in a martial arts class shortly after being infected and had to walk with a cane for about five weeks. I was an 18-year-old old man! I suffered some minor hearing damage as a result of the infection.
Now as a 25-year-old, I have four grandparents who all have poor hearing. Neither of my grandmothers spent much time around loud things, but their genes don't care. Your hearing, like your sight, is gonna go eventually. In a world with cars, guns, tractors, concerts, jet engines, and screaming children, it's impossible not to incur some hearing loss unless you choose to live away from everything loud, and even then there's no telling what your DNA has in store.
I'm not saying you shouldn't use hearing protection. You'd be a fool not to use it as much as possible given the available evidence. Just try not to do anything excessively stupid, and acknowledge that life is incredibly loud sometimes.