For general use (plinking, informal target, defence), you just can't beat the 9x19. With current generation premium JHPs, it is every bit as effective for defence as the .40 S&W with the added benefits of lower cost practice ammo, higher capacity and considerably faster recovery time between shots. It's also a lot easier on the weapon--.40 S&W weapons have more problems, problems earlier and more severe problems than their 9x19 equivalents. Thought it's very, very slight, the 9x19 will have a reliability edge over the .40 S&W.
Now, if concealed carry is in your plans (or you want to ensure domestice tranquility--all ways a big plus), the .380 may be a viable choice. You certainly give up some effectiveness by going to a .380, but as has been pointed out, it beats having nothing. A .380 will be a whole lot easier to carry concealed than an XD. You can drop a .380 in the pocket (or the wive's purse, etc. If you plan on carrying an XD, you are looking a good belt and a good holster or two at a minimum, and having to dress around the weapon. The .380 definitely has the edge when it comes to casual concealed carry.
An alternate solution, would be to purchase Makarov (not a .380 but close enough) for the wife for about $150 and the XD9 for you. That way she has her ".380," and you have your XD9 (and a weapon you can drop in your pocket or in her purse).