My personal experience has been that there's not a whole lot of difference. The kydex tends to wear fairly heavily on the parts where it holds the pistol in the holster (i.e. the front of the trigger guard) and relatively less on the rest of the pistol.
Leather tends to more evenly wear the finish on edges more evenly, not in the "hot spots" that tend to pop up with kydex.
I've heard, however, that kydex is usually a little easier to keep clean (wipe out dirt & sand) so it should theoreticaly be a little easier on finishes.