No point in considering the absurd. If the magazine will not feed the rounds, then it's no longer a magazine within the definition of CPC 16740 (although it would still be a magazine if loaded with .40 S+W).
As to the question of loading a .40 S+W magazine with 9mm, and assuming that you have a weapon that will fire that combination, you get into the issue of statutory construction and that brings in the intent of the law. The likely outcome (and I say that having taken a ton of cases to trial, but not being able to cite a case law directly on point) is that if you load the magazine with .40 S+W rounds, you're cool. If you load it with 9mm rounds, and have a weapon that will fire that combination, then you're in violation. The idea being that you're doing something contrary to the design of the magazine and that the apparent explanation for the departure from design is to create a magazine capacity of greater than ten rounds.
There is a good case law comparison involving stabbing instruments. Consider whether a piece of wire can be a "Dirk or Dagger." There is no law against carrying pieces of wire, electricians do it all the time, so do folks with iPods and earphones. But in the California case a People v Cabral, Cabral carried a piece a wire that he used to stab a fellow guest in a California prison. In upholding his conviction for the possession of a "Dirk or Dagger", the court did not look at what the object "was" (as you and the OP seem to be doing here), the court rather looked at how the object was "used."
By the same token, if you use an otherwise legal magazine, in a manner where it exceeds the legal capacity, then the DA is probably going to be successful in fanging you for the effort.