I still haven't opened my first can of APP yet, but I'm still shooting some of its predecessor, CleanShot. I've noticed that white stuff and checked it out again just now in my gun barrel from which I discharged my hunting load yesterday. I didn't find it to be varnish like after 1 100 grain shot, but rather powder like and soft, and a very tiny bit tacky or chalky to the touch, sort of like an extra fine corn starch, dextrose or lactose. I personally haven't shot enough of that powder at any one time to worry much about any wear and tear on any of the barrels used. The thought did cross my mind at one point early on in it's use, but I soon dismissed it.
But then I'm shooting either patched round balls where the lubed patch is contacting the barrel interior, or [usually] sabots with it, where plastic is meeting the bore, and swabbing can be done between shots if there was a continuing concern about it.
If a person shoots bare lead (like in a revolver or shotgun), or bare copper jacketed bullets (like in a centerfire), I would speculate that more friction and barrel wear could be caused by those harder/denser substances than by this residue.
A lot of Russian ammo now has "bimetal" jackets, and for a time I did think that material might create more barrel wear than lead or copper even though its supposedly softer than plain steel bullets, but millions of people shoot the ammo and never have I heard complaints about any premature barrel wear associated with their use.
There are acids, salts and corrosive biproducts in black powder too, and that doesn't stop too many people from shooting with it, while others highly praise it.
There's always going to be trade offs when choosing one product over another. I try not to use metal bore brushes to clean my guns either unless I think it's necessary, because they can create extra wear in some circumstances. It's the same with metal cleaning rods, ramrods and other things that one allows to enter their bore.
So there are choices to make, and you have made yours based on your gut feeling, which is your freedom of choice.
I wish there was a scientific report to indicate just how harmful the residue actually is to gun barrels. But since most barrels seem to be made to last a lifetime (or more), I'm just not sure how bad the residue truely is.
I don't really think it would "pit" a bore. At the worst, it might cause something like what running water does to rocks over millions of years, make them nice and smooth, which is going to happen after a countless number of shots anyway, whether it's caused by the powder, primer, bullets, cleaning regimen or a combination of all of these factors.