Since there's still no conclusive evidence that terminal performance differences due to caliber within the service pistol class have a significant effect in terms of actual shooting outcomes (the debate continues--it will end when there's conclusive evidence), it's going to be doubly hard to make the argument that different loadings within a given caliber will be a game changer.
What to look for?
You get a significant benefit from ammunition that expands properly vs. FMJ/non-expanding ammunition. It creates a blunt trauma effect which, although it doesn't reliably cause wounding, does provide a notification effect--it lets the person know they've been shot. This is important because that's not always the situation with non-expanding pistol ammo. People can be shot with FMJ during a chaotic encounter and not realize it until later. And we want attackers to know they've been shot because psychological stops are a major factor in the effectiveness of handguns for self-defense.
You get a significant benefit from ammunition that penetrates sufficiently to meet the FBI criteria. This makes it more likely that a loading has the ability to physically disable an attacker. If the attacker is disinclined to give up after being shot, or is chemically altered to the point that their mental functions and pain sensations are dulled, you need a loading that can go deep enough to get the job done.
You get a HUGE benefit from ammunition that functions reliably in your firearm. Does it feed, extract and eject properly? Does it jump crimp or setback? This means you will want to be able to buy enough of it to actually shoot a decent amount of it through your carry gun(s). This is even more important if the loading is unconventional in some way. I would want to shoot a decent amount of of a defensive loading through each gun I planned to use that loading in for self-defense--maybe 50 minimum for something pretty conventional, 100 minimum for something unusual or unconventional. Want to carry a round with a weird profile that could affect feeding in a semi-auto? If you have one carry gun and a backup--that's a couple hundred rounds. A home defense gun using the same loading? Another 100 rounds. "These cool monometal rounds machined to look like a space satellite are super-cool and work differently than JHPs so they must be better. But I can't afford to buy 300 of them just for practice and function proving! That would cost more than I spent on my carry gun!" That's a good point--just make sure you are actually getting the correct point. If you can't afford to prove your self-defense round in your self-defense guns, then maybe you want to choose something else so that you know you have something that works when you need it to.
What to look for?
You get a significant benefit from ammunition that expands properly vs. FMJ/non-expanding ammunition. It creates a blunt trauma effect which, although it doesn't reliably cause wounding, does provide a notification effect--it lets the person know they've been shot. This is important because that's not always the situation with non-expanding pistol ammo. People can be shot with FMJ during a chaotic encounter and not realize it until later. And we want attackers to know they've been shot because psychological stops are a major factor in the effectiveness of handguns for self-defense.
You get a significant benefit from ammunition that penetrates sufficiently to meet the FBI criteria. This makes it more likely that a loading has the ability to physically disable an attacker. If the attacker is disinclined to give up after being shot, or is chemically altered to the point that their mental functions and pain sensations are dulled, you need a loading that can go deep enough to get the job done.
You get a HUGE benefit from ammunition that functions reliably in your firearm. Does it feed, extract and eject properly? Does it jump crimp or setback? This means you will want to be able to buy enough of it to actually shoot a decent amount of it through your carry gun(s). This is even more important if the loading is unconventional in some way. I would want to shoot a decent amount of of a defensive loading through each gun I planned to use that loading in for self-defense--maybe 50 minimum for something pretty conventional, 100 minimum for something unusual or unconventional. Want to carry a round with a weird profile that could affect feeding in a semi-auto? If you have one carry gun and a backup--that's a couple hundred rounds. A home defense gun using the same loading? Another 100 rounds. "These cool monometal rounds machined to look like a space satellite are super-cool and work differently than JHPs so they must be better. But I can't afford to buy 300 of them just for practice and function proving! That would cost more than I spent on my carry gun!" That's a good point--just make sure you are actually getting the correct point. If you can't afford to prove your self-defense round in your self-defense guns, then maybe you want to choose something else so that you know you have something that works when you need it to.