Holsters aren't really the culprit here.
The porous surface of leather will capture dirt, and plastics will hold dust due to static and excessive oil. It's not the material that causes the wear, it's contact with dirt trapped on the surfaces that wears the finish.
When two dissimilar surfaces come in to contact, the harder surface will wear first! Softer materials trap and hold abrasives that wear down the other. It's counter intuitive, but mechanics have been dealing with it for decades replacing rear main seals, etc. The seal doesn't really go bad, the iron crank gets grooved and leaks. The new seal is just bigger to hopefully take up the slack.
Clean the inside of all holsters regularly to remove the abrasives, and don't oil exterior surfaces of the firearm.
