You're using media with rouge?
You can dump 'em in a big bucket of organic solvent. Gasoline, lighter fluid, alcohol, etc.
Or you can switch to media using green compound. That red rouge is good because it's pretty much not effective at all on steel (your firearm), but a little alum oxide buffing compound residue isn't going to wear out your firearm. After thousands of rounds, it might eventually start to polish your chamber, so what? Red rouge is rust. It's highly chromatic. You'll find traces of it everywhere. On your cases, on the bottom of your socks, etc. Alum oxide is practically colorless. They add a little green color for identification, but it's not as "staining."
Or try adding some dryer sheets (possibly dampened in solvent or plain water) to your tumbler to help remove the excess polish in there that traps the dust.