^ The brass bore brush is not the issue - it's allowing the steel cleaning rod to come into contact with the muzzle crown which can affect the accuracy of the gun. Again, toxicity = dosage - an occasional light scrape every now and then is not a big deal, but when guys obsessively scrub the bore like they're going to turn the gun over to a Drill Instructor for a white-glove inspection, that will take a toll.
Another issue is that aluminum cleaning rods can get grit embedded in them, and if you have to push the rod through the bore with great force(e.g. pushing doubled-up cleaning patches) the rod can bow and cause bore abrasion mid-way down the barrel. Solution: wipe the rod clean between passes, and don't force it if it don't want to go.
A final consideration is that rimfire priming compound often contains ground glass to improve the compound's sensitivity to firing-pin-impact - the glass dust causes extra friction within the priming compound, thereby making it more likely to ignite when crushed by the firing pin. Trace amounts of the glass may make it down the bore, which might cause additional abrasion if scrubbed back-and-forth too vigorously - it's also possible that a layer of deposited lubricant from the .22 bullets can keep the glass particles from contacting the bore.
All in all, there's a lot of reasons to avoid scouring your .22LR's bore too frequently...