I think pocket carry of some kind is the answer to business casual. And I may be wrong but I think you can get all the way up to a compact .40 into a pocket carry situation.
I have the same problem of dress, in that during the week I have to be in a button down shirt and at least chinos. As long as the pants are of reasonable fit, don't have to be baggy, I've been able to pocket carry 380s (KelTec P3AT and Bersa T380CC), a compact 9mm (Kahr CW9), and even a Ruger SP101 snubbie in .357, all in an Uncle Mike's #3 or smaller pocket holster.
I've done some fitting and am about to plunk down and buy a Springfield XD40 subcompact (3" barrel), since I think it will also fit in a pocket holster of about the same size. I haven't held a Kahr .40 (CW40?) but you can get very small pistols in all the way up to .45ACP if you want. I like my Kahr CW9 and see it being a backup carry, but I'm really looking forward to upgrading to 9+1 of .40 from what I have now, in a package about the same size.
So, I think with the right choice of gun, you could probably get a reasonably powerful handgun to fit into a pocket carry holster. There are some other inventive solutions I've tried out at a vendor's table at the gun shows but didn't buy, such as the holster that slips down the front of your pants and is held up by a fake pager that clips onto your belt. Didn't do it for me, but they seem to sell a fair number of those so some people must find that it works for them.