I have heard this issue come up time and time again
Leather is an abrasive. Its a *mild* abrasive, but its an abrasive all the same. Over time, any sort of movement inside leather (or most other materials) will cause a wear on the firearm. Most folks dont' notice this, as the wear is very slight, but some steels seem to show it more then others. Unless you are lining your holsters with sandpaper, this won't effect anything functional on your carry piece.
Quick story:
I grew up carving stuff out of wood. My grandpa and I shared the hobby, and he taught me most of what I know--I now carve stone for fun.
When you go to sharpen a chisel, you start with rough, then finer grit, then finer grit, then even finer yet, until you get down to a fine, tiny thin burr along the very edge. It with either leather or denim that you lock down that burr. Without an abrasive coating on the leather this takes quite a while. Hand cramping time...but if there is something in the leather, it will "cut" or knock down that burr faster.
I have wondered about Kydex. It seems to have a very low co-efficient of friction (or is it high?) Anyway, it seems to not have the rough surface of leather.
Further: Some leathers (especially black) are cured by soaking the hide in chromium. Yes...the same chrome chromium. This allows the leather to cure very quickly and with a lot of stability. These tiny bits of chrome get into the nap of the leather, and can act as an abrasive as well. Look for "Vegetable tanned" or "Brain tanned." to get the safest leather for your holster.