Marshall & Sanow's stats - believe it or not? State your case here.

[WebHobbit]

Evan Marshall & Ed Sanow's One Shot Stop Statistics are used by many folks to help them choose which ammo to use in their SD/HD/LEO weaponry. Other's scoff at these figures dismissing them as BS.

Here is an exchange between me and a couple other THR members in another thread (which we nearly derailed):

About "One-Shot Stops":

Evan Marshall himself has stated many times in his books/articles that he DOES NOT recommend someone stop with one round. Common sense tactics call for you to shoot until the threat is no longer a threat. DUH.

He just limits instances of documented One Shot Stops as the only thing he records in his database since only that will actually help determine a rounds true effectiveness. It is a pretty narrowly defined type of shooting to make his database. He also only counts torso shots.

--WebHobbit

Web hobbit,

The problem with Evan Marshall and the focus on one shot stops is that it leads the uninitiated to believe there is some magic bullet or caliber. Just get this caliber and everything will be cool (which of course it won't).

The reality is that any of the defensive rounds of 36 caliber or greater work. And none of them should be counted on for a one shot stop.

People should focus on getting a gun thats reliable, that they can hit with and practice gradually building up speed to continuous rapid on target fire.

As I said in my earlier post and as you said, common sense would dictate that you keep firing on target until that target is no longer a threat. However, you rarely read a post on those lines. You can read a hundred posts on "I carry _____ because its better than ____ and I know it will stop the BG in one shot" or something similar.

--SurfinUSA

The problem use of S&M even as you describe is ludicrous. Their stats (and I use that term very loosely) do not even give you the information you evidently are attempting to derive from it--S&M's "data" is at best misleading, and more accurately totally deceptive.

An example of how S&M works:

They have 100 samples of Cartridge A. Of the 100 samples, 70 fail to meet S&M's rather nebulous "criteria." Of the remaining 30 samples 27 were unobstructed thoracic cavity hits that resulted in "one shot stops" by S&M. No other "torso" hits (S&M criteria) resulted in stops. According to S&M, Cartridge A is 90% effective.

They have 100 samples of Cartridge B. Of the 100 samples, 50 do not meet S&M "criteria." Of the remaining 50 samples, 27 were unobstructed thoracic hits and resulted in "one shot stops." In addition to the 27 unobstructed thoracic cavity hits, there were 13 other "torso" hits that also resulted in "one shot stops." According to S&M, Cartridge B is is 80% effective.

Which one, Cartridge A or Cartridge B, was really the most effective on the "street?" Cartridge A with 27 unobstructed thoracic cavity hits rated 90% by S&M; or Cartridge B with 27 unobstructed thoracic cavity hits and 13 peripheral hits resulting in a "stop") rated 80% by S&M?

Folks, that's how S&M work.

When you look at the raw data (which of course S&M won't let you see), I see that in reality on the street Cartridge A, which they rate a 90%, really worked only about 27% of the time, and Cartridge B, which they rate at 80%, really worked about 40% of the time.

Which would you rather have--the one S&M rated highest, or the one that worked the best?
--jc2

Thread is located here:

http://www.thehighroad.org/showthread.php?s=&threadid=19837&perpage=25&pagenumber=2

Here is a link to the latest S&M book:

Amazon -Stopping Power link

So what do YOU think?

 

[JohnK]

The reality is that any of the defensive rounds of 36 caliber or greater work. And none of them should be counted on for a one shot stop.

People should focus on getting a gun thats reliable, that they can hit with and practice gradually building up speed to continuous rapid on target fire.

That's pretty much my opinion these days as well, but I'd say "35 caliber or greater". That would include the 9mm (.355"), 38 special (.357") calibers and at the bottem the 380 ACP.

Marshalls reading is entertaining, but JC2 does make a good point, I'm sure their raw data could be interpreted many otherways.

Lies, damn lies and statistics. - Mark Twain

 

[Sean Smith]

Food for thought: the M&S studies can't tell the stopping power difference between some .40 S&W loads... and a .308 rifle or a 12 gauge shotgun. All are within a couple of percent of each other on the OSS scale. Think what you want, but in my mind if a study can't tell the difference between the effects of a 165 grain projectile going 1,150 feet per second, and a 168 grain projectile going nearly 3,000 feet per second (or a 385 grain slug at 1,900 feet per second), it isn't much of a study.

 

[jc2]

When you look at the raw data (which of course S&M won't let you see), I see that in reality on the street Cartridge A, which they rate a 90%, really worked only about 27% of the time, and Cartridge B, which they rate at 80%, really worked about 40% of the time.

Which would you rather have--the one S&M rated highest, or the one that worked the best?

The methodology I described is exactly how S&M figure their "OSS" percentages. The "stats," as they publish them, are totally bogus and meaningless.

"NO, it's unscientific, convoluted BS!" is far too generous assessment.

 

[Tim Burke]

OSS

If you are measuring "One Shot Stops" and then you discard from your database every shooting that required more than one shot, your data becomes so skewed as to be meaningless.
They may be measuring something, but I don't have any idea what, and I don't think anyone else does, either.

 

[Boats]

Methods aside, there have been many instances where the alleged OSS reporting department said the shooting didn't go down like M&S said it did in their books. I second the notion that any study purporting to equate OSS percentages between pistols, rifles, and shotguns is selling something not kosher.

I also second the notion of shooting until the target is no longer a threat.

Doesn't it strike you as funny that no one in the revolver world tries to equate .38 Special +p to a .45 Colt? These equivalency arguments between medium and big bores seems to be only an autogun phenomenon or an arcane debate between 5.xxmm rifle owners and 7.62 ones. Odd how the ones who argue most vociferously are the ones with the marginal calibers.

 

[stans]

I used to live and die by the writings of Marshall and Sanow, but I see things differently now. Ballistic gelatin is very uniform, human beings are not. Ballistic gelatin doesn't get all dressed up before becoming involved in a shooting, human beings do, at least most of the time. Ballistic gelatin does not use narcotics or stimulants, human beings often do. Ballistic gelatin does not have adrenalin, human beings do. So I now read the writings of Marshall and Sanow and use their data as a rough guide, but not the be-all-end-all gospel of firearms self defense.

 

[Mannlicher]

I use my own testing to determine which load I carry in a SD handgun. Publications like Sanow's are helpful in some respects, but I only carry what is accurate, and effective from my gun. No, I don't have a ballistics lab, but I do have a chrony, and some small ability to test bullet performance.

 

[chaim]

Yes M&S "One shot stop" percentages are flawed. Even if the methodology was sound there simply aren't that many defensive or offensive shootings in this country every year to get significant results once they use any kind of screening to screen out useless data. Then once shootings where for one reason or another (some quite legitamate) the data is useless are taken out and you then separate the data into different calibers and you can have some sample sizes that are extremely small. Sure, maybe .38spl, .357mag, .45acp, and 9mm might have enough data for statistically significant results but what about .32mag?

That said, all measures are flawed once you try to extrapolate the results into the real world. Expansion tests in playdough, phone books, ballistic gelatin, or if you have some money in brisket? Even the best media won't duplicate human flesh, and when there are shootings involving people there are dozens of other factors as well to complicate things. Muzzle energy, sure you can use it as some kind of comparison but it still doesn't tell you how a particular round will perform in real life.

Personally, I think all major comparisons are useful to a small degree to compare calibers and rounds. However, the limits of these different methods must be considered. It is interesting, it can have some value, but one shouldn't make their choice based solely on one, or all, of them.

Pick your gun first on how it fits you and how reliable it is. Then using whatever method you prefer (one or all of the above, general reputation, gut feeling, etc) pick what you think will be effective. Anything from 9mm or .38+P and up will serve just fine, .380 and .38spl will usually work admirably, and even some smaller calibers in good loadings can work for you. Also, keep in mind that a .22lr is better than no gun at all. If you may need a gun and have the money for a used .38spl revolver, or a new Bersa .380 or a Pheonix .22 get a gun, learn to use it, then if you aren't comfortable with your caliber save for something better. It is best to have something, even if it isn't the "best" as you see it. If waiting for "the best" means going months or even just weeks longer with NO GUN at all, don't wait for the "best" GET A GUN, then you can get what you think it the "best" later! It is in those kind of situations, where the numbers from one method or another convince someone that a certain caliber is poor or worse, that they do a disservice.

 

[Handy]

Garbage in, garbage out.

In selecting only shootings that involved a single torso hit, M&S have started off with the most corrupt data imagineable.

It doesn't even matter if some of their other methods are questionable. From a statistics standpoint, they were doomed to fail.

The fact that some of their conclusions MAY match with reality is immaterial. A monkey might make some good guesses about the stock market, but that doesn't make the monkey Alan Greenspan.

 

[HankB]

1. Have M&S ever published or otherwise made available their raw data for public, independent review? If not, their "study" does NOT qualify as "scientific" - at best, it's anecdotal.

2. The most savvy, skilled police officers will, in the heat of battle, shoot multiple times as a matter of course, not waiting to see if the first shot fired was a "stopper." These instances will not be included, if I'm reading M&S correctly.

3. When M&S first started reporting OSS percentages, IIRC the .357 Silvertip had a higher OSS percentage than the .44 Magnum Silvertip. So a larger diameter, heavier bullet that was moving faster . . . was inferior? I don't think so.

My conclusion? M&S make for interesting reading, but their published info shouldn't be regarded as gospel. Today, Super-Vel isn't the only company making quality self defense ammo - EVERY major maker is. And I think people who use M&S statistics to switch from, say, one brand of 125 JHP .357 to another because of a few M&S percentage point differences are fooling themselves.

 

[chaim]

I said:

It is in those kind of situations, where the numbers from one method or another convince someone that a certain caliber is poor or worse, that they do a disservice.

HankB said:

And I think people who use M&S statistics to switch from, say, one brand of 125 JHP .357 to another because of a few M&S percentage point differences are fooling themselves.

You know I didn't even think about that. Let me modify what I said to

It is when people either wait to get a potentially life saving gun to save for another one because the other one has a few more points in stopping power, or when someone changes ammo for no other reason than a few points, that these measures do a disservice.

 

[Standing Wolf]

I think nothing short of an atomic bomb includes a guaranteed one shot stop. As well as I understand these things, the three critical factors are bullet placement, bullet placement, and bullet placement.

 

[Handy]

Chaim,

I dont' think you're getting the fundamental problem. Say M&S are comparing .45 and 9mm (this is just hypothetical):


They look at 100 shootings each. Of those, the heavier recoiling .45 creates 75 occasions were only one hit is scored. Of those, 55 were "one shot stops". The other 25 were two shot shootings, no stops.

The 9mm has only 6 occasions where only one hit was scored. Of those 6, only 4 were stops. The other 94 were two shot shootings, all stops.

So now we're comparing 75 of one to 6 of the other, and we've thrown away all the other shootings. The .45 has a better "one shot stop" record. Is it a better stopper?

 

[surfinUSA]

Web hobbit, Thanks for quoting my post.

Johnk, I was including the 9mm and 38 just rounding them up to 36.

My thinking is that a certain amount of caliber vs caliber and gun vs. gun discussions are informative and entertaining. But the underlying fact is that whether it is self defense or hunting, the tool is just a small part.

Marshall's work is flawed but not irrelevant. As an example we had a officer get in a gunfight with an armed robber. Of the 13 shots fired by the officer 11 hit the BG (unbelievably good shooting from what I've seen on police shooting scenes) ending the fight and killing him. Some said the 9mm hydrashock was ineffective, taking so many rounds to drop the bad guy. However, the lead homicide investigator showed me an autopsy photo of the guys liver with a hole in it almost large enough to put your fist through. These rounds worked just as designed.

Two things 1. some folks don't drop right away no matter what the damage and 2. This guy was probably dead before the officer stopped shooting and he hit the floor.

Just because a shooting isn't included in Marshalls work dosen't mean the round didn't work.

 

[WebHobbit]

Interesting points made by all. But you know there MUST be something to them as many of the rounds that have the highest S&M percentages also have an awfully good and respected street rep - like the 125 grain .357. No one doubts the effectiveness of that round. Part of their methodology is to take a round that has fantastic real world performance on the street & test it in Ballisitc gelatin. They then take all the measurements: wound channel, cavity size, penetration etc. Now they fire other rounds that don't always have much of a track record and test it in gelatin.

Is not reasonable to draw some conclusions from these comparisons?

 

[Sir Galahad]

If you want to read the work of people that actually KNOW what they're atlking about, check out the work of International Wound Ballistics Association, Dr. Martin Fackler, and David DiFabio and his ammolab.com site. David DiFabio has done some excellent work and you can ask him questions. If he doesn't know about a certain load, he'll say he doesn't know. He won't guess or make something up.

"One shot stop" is fable. I bet one broadhead arrow would kill a guy by massive hemorrage, but not before he could kill you. And that same arrow in the kneecap will stop a guy. But that doesn't mean a bow makes a good home defense weapon. M&S use the old P.T. Barnum method of showmanship to peddle their flawed data. "One shot stop"? No such thing. Some people will drop immediately when shot by a .22LR. Some will not drop when shot by a .45ACP. If we're talking massive tissue destruction as in a shotgun, yes, we might see "one shot stops", but we cannot state that as fact. Confidence and marksmanship are what make a good defense, not the weapon. Like the man said in the old Westerns, "Hold your fire til they get close, son, then aim plumb center. Noise don't kill rustlers."

 

[Handy]

Hobbit,

I refer you back to my monkey analogy.

 

[cratz2]

Good reading, maybe even worth sparking a couple debates with your fellow shooters, but I can't give their data analysis or reporting much credence.

 

[355sigfan]

Evans a good man. His work has some flaws. But its valuable as a guide. Is it the gospel no. The fact that his opposition hates him so is proof that there is something to his work. His opposition has yet to do a study for themselves correcting the flaws they feel are present with Marshalls work. Until they do some real work and leave the lab they will not have any credibility.
PAT

 

[Hal]

It's input and information.

Speaking of OSS and lack thereof.

Few weeks ago I was watching some footage on CNN from Iraq. 2 Iraqis were shooting at some US troops @ ~ 500 or so yards and running between 2 buildings between shots. Our guys decided to take them out with what looked to be a Javelin. The first Javelin failed to fire. ^The second one sure did the job though. Both buildings went up in a very impressive way.

Man, all I could think of later was that our guys should have showed that video of what a Javelin can do to a tank to the Iraqis, then offered them $10K each to drop their guns and catch a plane to Atlantic City to open a slurpy shop on the Boardwalk.
@ $100K for the tube, and $60K for each Javelin, we would have saved a bundle.

I guess what I'm saying is Standing Wolf is on the right track.
OSS = Thermo-nuclear
Anything else, shoot twice.

 

[Handy]

355sigfan-

There are people who say the holocaust didn't happen. Their "work" is universally dispised. I don't think that's proof that there is "something to [their] work".


As to doing better, the M&S study provides a case study for the pitfalls of anequedote based statistics. I think the experts realize that the number of shootings vs. the number of variables produce a situation that is beyond such analysis. M&S, having no stat background, simply didn't realize what they were getting into.

No statistical scientist would touch this one with a ten foot pole.

 

[Sean Smith]

No statistical scientist would touch this one with a ten foot pole.

But that's why M&S have their "street cred"! They aren't a bunch of pointy-headed lab nerds, and so what if they don't worry about doing the math right?

Jeez, people, THINK for a minute...

1. If the study was any good in principle, it would still be worthless because the sample sizes are so small,
2. We already know the study isn't any good in principle,
3. We know the outputs of the study, regardless of #1 and #2, are completely bogus on the face of it, since it can't tell the difference in on-target effects between .40 S&W and 12 guage slugs.

What's left in their defense? That they are "street cops" instead of "lab nerds"? Using anti-intellectualism to justify a statistical study is like using Amish theology to argue the virtues of a Ferrari.

 

[355sigfan]

Well were going to have to agree to disagree. The funny thing is that the 2 camps agree on most loads with the exception of the 147 grain 9mm. Which by the way you could not pay me to carry.
PAT

 

[Mike Irwin]

Interestingly enough, and despite the pervasive and demonstrated flaws inherent in Marshall & Sanow's work, if you look at its recommendations and those of Dr. Martin Fackler, you'll see that a lot of the same loads place highly for both groups.

With Marshall and Sanow I tend to see that as a case of the "truth" burning through the terrible flaws in their methodology.