I call it possible but unlikely.
I think it is difficult to say that a change is bad just because it is a change. A better trigger doesn't mean a BAD change. If it was a cheaper, lighter, or defective trigger, I could see someone trying to call it a 'negligent' modification. I put different grips, sights, and guide rod on my carry gun, I think a DA would have to be pretty creative to say that these changes were negligent in any way.
The main thing I tell my carry students, is not to ever disable any safety device. Don't pin down grip safeties, don't disable magazine disconnects, things that actually change HOW the gun functions.
You can 'what if' all day long, and if a rookie DA is REALLY trying to hang you up, he will invent a silly reason. I think this far enough down the list of real reasons to mess with you as to not worry about it.