I personally won't go any smaller than the Kahr line of pistols, that size is the smallest I would carry at this stage. I had the seacamp, for 15 years and honestly it was too small to get to, unless you were standing and it was in a OWB holster. I almost never carried it, unless it was a dressy affair, and nothing else worked with a suit.
Once you drop that size gun in your pocket, it takes more time to find that darn thing and get it out, plus the controls become difficult to manage under pressure. I think that a carry gun should be easy to get into service, quickly as your life depends on it.
Even the old Jetfires and Baby Browing's are difficult to get a hand on sometimes because they are just too small, "forget weather you like or don't like the caliber", the size becomes a problem when you are in a tough situation and need that gun right now. my Pm9 is right at the edge of where I am willing to go to sacrifice size for concealment.
If you pocket carry , like in a Remora, type holster you understand that drawing the gun, takes practice, to clear the pocket without getting hung up on the way out. Even an IWB can be tricky with some small guns, depending on one's weight and clothing.
If you have a gun that only takes a 2 finger grip, it is harder to grab it under pressure without an extension on it. Just another thing to consider.