I had a college professor once tell me...if you can't make up your mind, flip a coin and register your first thought when you see the results. If you're disappointed when you see the coin, then that indicates what you really wanted the outcome to be. If you're happy with the outcome, then you also know what you really wanted to start with. If you're indifferent, then you really don't care, and if it involves spending money then you probably shouldn't.
As stated by the other poster above...go shoot several .40 guns, either friends' guns or range rentals. You will make up your mind without forking over the dough for a new pistol. Then flip that coin if you're really not sure.
FWIW, I bought a SiG P226R in .40 as my first handgun 5 months ago. I went .40 at the recommendation of 2 friends who are LEOs. I now have 3 pistols, that original P226R, a P226R stainless, and a P239SAS all in .40. I have kept buying .40 mainly because it makes ammo shopping and handling easier. I'm a big, heavy guy so the snap of the .40 doesn't bother me. But the ammo is NOT cheap.
If I had it all to do over and know what I do now, I might buy 9mm instead of .40 (and still pick up those .45s), because the 9mm is cheaper...nearly half the cost for range ammo compared to .40...and easier to shoot and still very capable for self defense.
In the past 5 months I have have also shot other guns that belong to friends and a couple range rentals, covering .357sig, .45acp, and 9mm. I have no doubt I am going to buy a 1911 just for the fun of it, and may pick up a SiG P220 in .45 as well. But with the .40 pistols that I already have, I have no further need or desire to get a .357sig or 9mm.
Getting a 9mm now would not increase defense purposes of my collection, and the cost of additional pistols or even the 9mm conversion barrels to go into my existing guns hardly justify the savings on ammo. It would take 2000 rounds of 9mm to justify the cost of one $200 BarSto conversion barrel.
Seems to me you are already committed to 9mm and .45 and perfectly happy with them. I doubt a .40 is going to make you any happier, but it's going to lighten your wallet a few hundred dollars easily. And it's not really going to improve your self defense picture any. Why bother???