There is no such thing as too small if it is the only thing available. A .22 LR at your side is infinitely more capable in an emergency than a .44 Mag in your safe. But it is not just caliber, it is whether the gun is too small or too large to make it easy to operate. To answer your question in short, I'd prefer to grab a duty-sized semi-auto, or maybe a medium frame revolver.
More extensively: Personally, the only handguns I have smaller than 9mm are 5.7 FN and 7.62 Tokarev, and, hypothetically, they will both give somebody pause when you're piping lead melodies out of the bang flute. I'd feel most comfortable, though, with 9mm Luger or larger. The only reason I am not fine with the Tokarev round is that they are usually older model guns (mine is a CZ 52) that aren't conducive for self defense. If there was, say, a Ruger SR762 chambered in the 7.62x25mm, I'd probably be fine with that.
At this time, I don't really plan on purchasing pocket pistols chambered in .380 ACP, .32ACP, .32 NAA, .25ACP, or .25 NAA for defense. I might like to have them just to have them, but not as defense guns. There was an experimental 7.92mm (.312") caliber based on a .30 Carbine, but shortened to work in a semi-auto. If that was available, I might even be willing to use that. A .327 Federal mag, even if loaded with .32 H&R Mag, would do if the frame was large enough to make shooting easy. But pocket pistols are, to me, last ditch attempts at self defense that are only fitting if the only place to conceal carry is in your Speedo.
Even if the pocket pistol is in .38 Spc or .357 Mag, the diminutive size makes operation difficult, and in the case that the first shot doesn't drop the attacker or scare him off, I would be spending more time fumbling around with the thing than I would taking shots. My big boys, like the .460 Mag, have a similar problem. They are more effective on paper, but they are unwieldy, and you can't miss a person enough times to make them stop. One hit with a 9mm is better than 5 misses with a .460 Mag. The biggest I'd go is 10mm Auto, .357 Magnum, or .50 GI.