Yes, refinishing it will harm the value, except in rare cases where the finish is almost completely gone.
That said, you will never recoup the cost of the refinish, regardless.
My rule of thumb is I refinish guns only when the continued use of the firearm in the manner I wish to use it will, or is likely to, result in actual damage to the workings of the arm. I have a Sig P220 that is close to that now. Almost no finish left on the slide from years of carry in all weather conditions.
That said, a little holster wear on a DS I see as a badge of honor from actual use. I would not refinish such an arm.
That said, it is your gun to use and enjoy how you see fit. If you see it as an investment, do not refinish. If you see it as a tool or toy, do what will bring you the most use and pleasure.