Grit in the works.
Very few of us pull a trigger straight back, which is what triggers are designed to do. Most of us pull slightly to the left or slightly to the right, depending.
I'd guess your gun has some dirt or lint somewhere in the vicinity of the trigger, more on one side than the other. If you strip it down to the frame, clean everything within an inch of its life, oil it and wipe off the excess oil, and carefully reassemble it, you'll probably make two discoveries: the gritty feel is gone, and b.) the trigger (both single and double action) feels smoother.
That saidâ„¢, it's possible your gun has worn or misfit parts. I believe it's much more likely it's just dirty, but there's only one way to find out. My pre-agreement model 60 was filthy inside when I bought it. Cleaning it thoroughly turned a pretty good trigger into an excellent trigger. Since I frequently drop it in a pocket, I run a patch or two through the barrel every week, and strip it down to the frame for a complete cleaning every six or eight months. It's amazing how much lint ends up in the works.