The OP was talking about quitting over fear
"Fear" is an emotion. A reasoned calculation that the risks of an activity now exceed the rewards is nothing to be jeered at, though.
I don't ride motorcycles. It's not because I have a negative emotional reaction to them... it's because I think the inherent dangers (including the dangers posed by other drivers whose action I cannot control) outweigh the enjoyment I might get from it.
For activities where our own actions are the primary influence on the level of risk - and reloading is certainly one of those activities - it is reasonable and prudent to periodically make a conscious assessment of whether anything has altered our ability to control/mitigate risks to an acceptable level.
I have seen a few older shooters "retire" themselves from USPSA/run-and-gun games, because they find themselves having a harder time maintaining the spatial orientation needed to avoid breaking the 180. The games that were once acceptably safe for them became unacceptably unsafe. I have a lot of respect for those guys who take themselves out of the game before they hurt themselves or someone else, rather than waiting to be told they are no longer welcome as participants.
I'm in my mid-40's and I hope to have many, many years before I have to really wrestle with these questions; I also hope I will have the integrity, honest self-awareness, and presence of mind to make rational decisions when the time comes.
I think it's unfortunate that the general anti-gun climate/pressures have made us all
so defensive of the
right to engage in various gun-related activities (from reloading to concealed carry) that we've developed a culture of thinking
anyone can do these things responsibly. Well, that's not true. There are some people who have no business reloading. There are some people who have no business handling a loaded firearm. Thinking that the government is a poor arbiter of that question is
not the same thing as thinking that people should not answer those questions seriously for themselves. And sometimes the answer changes over time.