Never had an issue with ejection of 9mm when using a 9mm barrel in a 40 slide, but it would take you just about 1 minute longer to swap trigger housing assemblies while you were swapping barrels or barrel/slide assemblies.
As noted, not recommended for carry when using the 9mm barrel in 40 slide. The additional breach face width can leave too much room for the case head to move around as the slide is cycling to the rear. More of an enhance potential for trouble than one that has actually been seen much, but it is a real possibility. Better to use a complete top half than to mix parts.