I was trained on the 1911 in the Army Reserves, and carried an M9 on duty. It's not really much about speed during the encounter - it's about handling the weapon in a reflexive manner that doesn't require thinking. If you are drawing a pistol, you don't stop to do each action one at a time, you do them in an overlapping manner during the draw so that when the weapon is up, the only thing left is to decide to shoot. In self defense, that decision should have been already made - you don't brandish your pistol in public to just make a point. You make it go on point and pull the trigger, done.
Therefore, flipping off a safety as part of the draw doesn't necessarily make it an overly complex, slow procedure. Especially if you shoot a gun with one regularly. If fact, if a particular gun is your daily carry, and self defense the primary role in your training, you practice only with a gun with similar controls. And you practice with a larger firearm because it's easier to do with extended ranges sessions. If shooting a smaller one creates a distraction because the recoil is snappy - and the LCP IS snappy for a lot of people - then don't practice with it. You will only reinforce bad habits.
LCP as a carry, practice with a Glock - and maybe add the NY ten pound trigger, too. Make your larger practice gun the same as your CCW. Then you have nearly the same inputs and no surprises or confusion.
As for pocket carry with a single action cocked and locked, it's very much up to the individual, and what you are comfortable with. I trained early on with a safety, my personal gun is a Glock 19C, and I CCW an LCP. Will it go off in my pocket, no, not in a holster. None the less, I dislike carrying it with no safety at all. I'm going to sell it and get a P938. The caliber is more effective, the size increase isn't much. Certainly bigger than an LCP, but compared to a P238, you have to look twice to see it. Bit longer barrel shroud, some in the grip. Incremental at the most.
If carrying a single action with safety doesn't sit well, better to admit it and deal with it. Carrying no safety at all is just as distasteful to others. The gun still won't fire until you put your finger on the trigger. If a few microseconds of speed mean life or death, I have to say, most of us would have moved out of that neighborhood years before. I find that discussion one based on the old locker room measuring contest, more hutzpah than reality. What is important, bar none, is that you carry, and if one or the other keeps you from it, then better to have one than need one. Just practice with the same action to keep your skills up.