I like the 9mm, as it's reasonably effective, cheap, and there's a plethora of choices for practice & self-defense cartridges. Granted, most 9mm semiautos hold no less than 12 rounds, but semiauto handguns in larger calibers are catching up in terms of capacity. The Springfield XD .45ACP is a good example. The 9mm Parabellum is a good choice for self-defense, especially with modern high-performance JHP & EFMJ (expanding full-metal-jacket) rounds, and the variety of 9mm semiauto handguns to choose from is probably larger than any other caliber pistol. It's still a fairly fast round, and over-penetration is a distinct hazard to be aware of.
The .40S&W is actually my second-favorite caliber for self-defense with a semiauto. The FBI switched from 10mm to the .40S&W after tests showed that the 10mm was a little too powerful. FMJ is generally a little more expensive than 9mm FMJ, while quality .40S&W JHP is usually about the same as 9mm JHP.
10mm would be my next preferred semiauto cartridge, and it's been gaining popularity, with guys like Ted Nugent choosing 10mm as their favorite. And IIRC, some police-departments allow it as an optional caliber for officers. It's a very powerful cartridge, and I'd be careful when using it for self-defense due to over-penetration. Expensive too, not something I could afford to practice with like I do with my 9mm.
I do like the .45ACP, but JHPs can suffer from a lack of penetration & expansion when shot out of short barrels. Still, it's not meant to be a light-speed round, and it's one of the few calibers that FMJ performs well with in a self-defense role. .45ACP is kinda pricey compared to 9mm & .40S&W.
I don't know much about the .357 Sig, and I'm not a big fan of the .380 as a primary carry caliber.
But my all-time favorite semiauto self-defense cartridge is the 9x23mm Winchester. I guess it's considered to be an "exotic" caliber, it's kinda hard to find commercially and is expensive, but it has .357 Mag performance in a 9mm package.