Respectfully, Uniquedot, you are confusing the action of dirt and grit being carried by a soft substrate with the action of the substrate itself.
Perfectly clean water will not erode rock by wearing it. Water carrying small pebbles, sand and particulate will erode rock.by wear from the solid matter it is carrying, and water may slowly dissolve minerals in the rock due to chemical reactions.
Clear water is not abrasive. Think of a fish tank power filter...water runs though the soft plastic for years with zero wear...the plastic is not even polished.
The cotton of your pocket, on a microscopic level, contains contaminants that make it an extremely fine abrasive cloth. ( I am not insulting the cleanliness of your pockets!)
The most common definition of what relative hardness of a material is talks about the ability of one material to scratch another. Lots of scratches eventually make wear. If a material is softer than another, it cannot scratch it, hence, no wear.
Bob