100% this.
Primary failure mode on bottleneck rifle cases is case failure at the webbing (case head separation), as brass flows under pressure while fired, and again is squished when sizing. What you are seeing as growth is the brass being forced against the chamber under tremendous pressures. Eventually, after enough firings and sizing, this leaves the brass around the case head thinner. What was previously material in the shoulder is in the neck, what was previously material in the body is now shoulder, etc. The material you remove while trimming has to come from somewhere.
Another failure mode is shoulder separation, if you have a loose chamber and you are constantly having to push the shoulder back, it can create a weak spot at the shoulder/body junction and then you have to break out the handy-dandy broken shell extractor tool to pull out the shoulder and neck, when the rest of the body is ejected.
Personally, I *do not* attempt to push the shoulder back whenever possible. I match bolt action brass to that rifle, and that is paired for life. I only neck size from that point on. I can get *many* neck sizings out of that brass, as there is never a need to bump the shoulder back, sometimes 10 or more, before there's any sign of failure, and I don't have to trim as often. When it comes time to need a third trimming, it's recycled.
For 5.56 where I don't know what rifle it'll get shot in, I use a SAAMI small base sizing die to ensure it'll work in all of my rifles without issue, since it's random luck when that cartridge gets put in to what rifle. This works the brass far more than neck sizing and does come at the cost of that recycling point of the brass happening much quicker. But this is because each time I'm resetting the case back to original SAAMI specs, and not just neck sizing.
If you are shooting *that* ammo in *that* rifle, there's not a compelling reason to ever so much as touch the shoulder. I have had cases go through 8 or 10 or more firings in bottleneck rifle (through that same gun) without once touching the shoulder during the resizing process. That is due to that brass being fireformed to that specific chamber; it greatly extends the lifespan of that brass.
For semi-autos, though, you do want to at least full length size or (if you have multiple rifles) pick up a small base die to get them clear back to original SAAMI specs. Cases won't last as long but reliability is a bigger concern!