This is a difficult question to answer for me. So far, every firearm I've acquired has been one I want to keep. I've made good choices, both new and used. Near the fringe end of the spectrum of keep/sell is the S&W made Walther PPK/S. Had to have it for the Bond appeal, but it's finicky. Oftentimes a few shots in, it catches a round in the slide when cycling from the magazine and jams the action requiring a tap-rack-bang. I replaced the spring with a lighter one, lightly polished the feed gate and tried a few different mags. I've found it runs fine with Speer Lawman FMJ in a Walther OEM mag. Not ideal - every other firearm I have seems to eat everything. But I still want to keep it.
I appreciate the plastic fantastics for their utility. My Glock 19 Gen 4 has been reliable and I trust it fully. But it surely isn't one of my prettier pistols. Also a 1911 guy - I shoot each differently; the difference in grip angle isn't really a pain point for me. Point with my trigger finger when shooting a 1911, point with my thumb when shooting a Glock.
I don't like the internal lock on newer S&W revolvers, though I have a few. The key safeties haven't locked up on me, and I don't anticipate they will, but I'd rather not have them. Some of my buddies installed a plug to get rid of them on theirs.
My Rossi R92 in .357 mag has a safety on the bolt. I ordered a plug to replace this; but I like the rest of the rifle enough to not be bothered with the modification.
The frustrations I have with these items are addressable. I enjoy working on my firearms that aren't dedicated to SD, but I understand how these could be turn-offs.
Biggest barrier to me is probably a bad trigger on a defense-oriented firearm. I try to keep those with stock parts, and some pistols (and rifles) just feel too mushy and unresponsive.