What's 'wrong' with .40 S&W? Not much, when it comes right down to it. Just as with other service calibers, it's more about placement and whether critical tissues, structures and organs are hit (and how quickly?).
Back in '07, SFPD officers had to use their issued .40's against a 243-pound Siberian tiger, named Tatiana, that escaped her enclosure and went on the attack.
It was later determined that Tatiana had been hit seven times: twice in the head and five times in the chest.
I'd just attended a LE firearms instructor update at SFPD, maybe a year previously, and they were carrying 180gr JHP's in their .40's. I dunno what specific load was being carried by those officers in '07, but I knew they'd previously issued Rem GS and Fed HST, and were in the process of wanting to change over to something else when I was there. The rangemaster said he usually spent $750K for his annual ammo budget at that time, and they frequently changed ammo for various reasons, if only to prevent some ammo company from thinking they had a lock on the PD's business (paraphrasing him). Sure, there were some previous experiences he related of some key-holing bullets, low-powered loads (squibs) and some puzzling accuracy issues (and what he suspected was a barrel manufacturing issue with a previous gun company's weapons), but while those things are relatively infrequent (rare-ish?), if you buy enough guns and ammo and shoot enough rounds, you can come across those sort of things over the long term.
I remember in earlier decades when I took my family to the SF zoo, or chaperoned a group of kids for zoo field trips from the elementary school our kids attended. I always carried a full-size (4") .357MAG revolver concealed, because of the 'remote' concern that a dangerous animal might escape containment. Go figure. Then again, that's what I carried on-duty in those days. There was a time many, many years ago when I carried a concealed Colt Commander to a different zoo.
Bottom line? I'd not feel any particular concern or disdain for the .40/180gr combination as a general 'outdoor' defenisve load. It was one of the issued loads I carried when I worked some rather remote rural beats (also having carried issued .357MAG, 9mm & .45ACP in the same areas over the years). I used 158gr .38 SPL +P to kill seriously injured deer (replacing the .357MAG loads in my .357 for that purpose). That was why I carried the .38's in the car in those days. Others in uniform duty at my former agency have no doubt used .40, .45 and now the current 9's for the same role.
Just remember ... TANSTAAFL