does the material applied to the outside ever effect your holster material?
Not if you wipe it on, and then wipe it off with a dry paper towel.
Its the day's accumulated gunk that is the biggest threat to the finish.
The idea is, the Break-Free is a solvent that removes the day's contaminants from the surface of the gun. Wiping it off gets almost all of the "oil" off the gun, and removes the "emulsified" goo from the day. That leaves you with a CLEAN gun.
Holsters - especially LEATHER holsters - will accumulate contaminants that will eventually erode the finish on a gun. This is what we call "holster wear." It is unavoidable. If that gives you heartburn, I can't help you on that.
Needless to say, a holster worn daily is eventually going to wear out. Just like your belt and your shoes will eventually wear out. Most holster-makers recommend carnauba wax on the outside of the holster, and a TINY amount of silicone on the inside of the holster. The silicone should be applied very lightly on the gun once in a while, and never applied directly to the holster.
Kydex holsters may not absorb contaminants as bad as leather, and may last forever, even with daily use. They do seem to eat gun-finish about the same as well-maintained leather. I don't think Kydex looks very stylish, mature, or sophisticated though.