Depends on the semi. While I often carry a Smith Model 60 in .357, the coiled mainspring makes the DA trigger a lot less user friendly than a K or L frame. My 60 wears Pachmayr Compacs, which makes the gun more shootable than the usual boot grips, but they also make it larger.
Many small semis, usually in .380, have more recoil than bigger guns if they depend on a blowback mechanism rather than some sort of link system. Small 1911's are handy and easy to shoot well, but their reliability is dubious. Double action semis usually have a crappy first shot trigger pull, worse than a J frame's.
The upshot: Get a small Kahr, Glock, XD, or Smith M&P in 9mm. They're easier to shoot well than a J, likely to be more reliable than a baby 1911, and have better trigger pulls than TDA or standard DAO guns. They carry more rounds than a J, too. While they're blockier than a J, in a proper holster that probably won't matter much.
So why do I carry a J? They'll ride well enough in a pocket, unlike the above semis. If I use to a holster, I'll step up to a 3" K frame because they're so much easier to shoot well. However, the Smith M&P Compact is starting to call to me. Based on what I've read, their triggers and ergonomics put them ahead of the other polymer guns (Kahrs are too small for my hands), and they're a lot lighter than my favorite auto - the CZ 75B.