I worry about wearing out my handguns about as much as I do wearing out my vehicles.
It will eventually happen to some degree, at some point, if I use them often enough and long enough. However, I can mitigate the expected wear & tear with periodic inspections and preventive maintenance, as well as accepting the necessity of some parts replacement as time passes.
Now, my role as a LE firearms instructor and certified armorer allows me access to a 'free' ammunition inventory, as well as different issued weapons (based upon assignment needs, inventory rotation, turn-over, etc.). This means I do a fair amount of shooting from time to time. Not as much as folks of a competitive inclination, but probably my fair share for practice/training as a LE firearms instructor.
I've only run up more than several thousand rounds on a few issued and personally-owned weapons, but that's because I'm able to spread out the usage across a number of different ones.
I can think of several pistols I've picked up in the last several years which have been fired around 9-10K rounds. I can think of an issued model which I estimated was fired about 45K rounds (for training) before I rotated it out-of-service for another one. Yes, some parts had to be replaced during that time. Not unexpected. Most of them only received several thousand rounds before being rotated for others for various reasons. None were 'worn out' ... and the fellow who received one of them, through which I'd fired upwards of 8-9K rounds, was very pleased with it. It had been carefully maintained and was well broken-in by the time he received it. I sort of regret having turned it back in, but I went with a compact model for my plainclothes assignment and didn't see the need to keep the full-size gun at the time. Oh well.
So, while I don't subject them to wear & tear for the simple 'exhilaration' of it, but do so for training/practice purpose, I have tended to run up more 'mileage' on many of them than what might be considered 'normal' to be the 'average' private owner/user. Not surprising, considering that a surprising number of private owners may never fire 500 rounds through their handguns, let alone 5,000 rounds ... or may only fire a couple of boxes of rounds and then lose interest. Even some LE may never fire more than 100 rounds a year over the course of their careers. In such circumstances neglect may be more of a concern than accumulated wear & tear ...
The longer I shoot, the more I've come to remember to have a specific reason for each and every trigger press during training/practice. Not just burn powder for the sake of burning powder.
Of course, that's a carry-over from my years of martial arts involvement.
Bottom line? No, I don't worry about wearing out a handgun. It may happen at some point, some time ...
I'm more concerned with losing perishable shooting skills.
I can always buy more parts, components or even another handgun. I can't 'buy' skill development, refinement & skills maintenance, though ...
Guess which concerns me more?