I suppose it depends on if you get down into benchrest or other highly accurate pursuits, and you want/need to worry about neck tension. For varmints, I definitely anneal because we're often out to 300 yards on a 4" target. Or you pitch brass at some point.
I keep brass in lots and by fired count. I've pushed my .243 out to 5 firings, and I've found that's about a point where I start to have problems - not just neck tension but also extraction issues. I started annealing after 3, and that's working fine for me.
Of course, you may have different needs. I fireform and bump. If you full-length resize then that'll be different.
I will say I didn't anneal for years, but it sure solved some problems for me.