In addition to those times I'm target shooting or just "plinking", I prefer adjustable sights on any handgun that I carry holstered, whether carrying it for hunting or for self-defense. The only times I prefer fixed sights are those occasions when I may have to draw from a concealed/covered location (pocket, ankle, sob, iwb, etc.) or if I have the gun for its "historical" character (i.e., a sa revolver for use in cowboy action shoots, etc.).
Because I reload a lot and use different powder amounts/bullet weights in various configurations, an adjustable sight is the only practical solution for accomodating poa with poi when shooting different loads in the same handgun.
Finally, short of the severe use one might expect if a pistol is being used for military duty or SWAT team assignments, in my experience (which includes thirty years of le employment), adjustable sights aren't nearly as fragile as some people make them out to be.