Yeah, I have seen this idea go both ways.
I have personally seen guns rendered useless by people trying to "Fluff And Buff".
There are lots of little things about a gun that really have to be a certain way if they are to be expected to work as designed. The feed ramp in a 1911 .45 auto is a prime example of this. The size, shape, and angle of that feed ramp was very carefully designed by a man that most educated folks consider to be a genius. (John Browning)
Thinking that a person who has to ask "Will it hurt it?" on the internet is going to improve on That Inspired Design by removing metal and resizing, reshaping, and relocating it is a bit silly, isn't it?
Why don't we just re-write some of those old Edgar Allen Poe stories while we're at it, huh? Surely we can make those better too!
Of course, the other side of the story happens some times.
I have a Ruger Speed Six. That pistol has a frame that was made as a casting. The whole point behind that idea is to reduce cost by eliminating hand labor required to build the pistol.
So, there was this ridge on the rear of the frame that caused the lock to bind when you pushed the cylinder into the frame after loading the pistol.
If you pushed on the release button as you closed it, it operated smoothly, but it was difficult to close otherwise.
I carefully smoothed that ridge and now the cylinder opens and closes slick as butter.
So, Yes... sometimes grinding on a gun is a good thing.
BUT
It needs to be done by somebody that knows exactly why and exactly where and exactly how to go about improving the gun.
Of course, we are talking about YOUR gun, so do with it as you please.
Just think first and make sure of what you are doing because taking metal of is easy but putting it back on is not.