Why don't gun rags use standard statistics?

Status
Not open for further replies.

Bacchus

Member
Joined
Dec 24, 2002
Messages
612
The subject line says it all. Here's an example.

In the Jan '03 issue of American Rifleman, the authors of one article are comparing Hevi-Shot with lead and steel shot. The text of the article is accompanied by several tables, most of which make no sense and don't allow for any real comparisons.

For example, one table lists average striking velocity at 40 yards. Instead of calculating one specific number for Hevis, one for lead, and one for steel and then comparing them, the authors list both a highest and lowest velocity. That makes statistical comparisons impossible.

Whatever happened to good old fashioned averages and standard deviations? Then it would be easy to compute t-tests and/or ANOVAs on the numbers.
 
I imagine most gun rag writers were either out shootin' or sittin' in jounalism or composition class while some of us were trying to wade through statistics.

Your point makes lots of sense to me but I'd guess your talking about a skill set most gun mag writers don't possess?

RJ
 
Write them and let them know. Maybe your observations politely stated would bring about change. You never know unless you try!
 
While the Rifleman should be called to task by the membership for such sloppiness and might well be improved thereby, I hold little hope for the fully commercial rags. Years ago, Skin Diver, a Petersen publication at the time, published a glowing review of a SCUBA regulator in the same month that the US Navy Experimental Diving Unit ranked it second worst on the market, one of only two regulators of about fifty to flunk NEDU's testing and be disallowed for purchase by even recreational services. Not long after, I met a Petersen's representative and asked about the contrast. The rep gave me a withering "Get Real!" glare and bluntly stated that, given their Skin Diver advertising budget, ALL of that manufacturer's products were simply wonderful. The realities of the business, the expense in time and money of a statistially worthwhile sampling process, and the scarcity of real statistical knowledge among the readership all argue against change.
 
The rep gave me a withering "Get Real!" glare and bluntly stated that, given their Skin Diver advertising budget, ALL of that manufacturer's products were simply wonderful."

First, I'm surprised that anyone admitted that to you.

Secondly, from my time at American Rifleman, I can tell you that for the most part fiscal considerations do NOT drive reviews. The majority of Rifleman's operating funds come from a line item in the general budget.

As for the statistics, some of it has always been giving members the most understandable statistics, ones that would hopefully make the most practical sense to them.
 
My first thought about it was in line with what you mentioned, Captain, that most people might not understand the specifics.

However, it seems like a disservice to present information in tables with the implication that legitimate comparisons can made on that data.
 
If they used standard statistics, it might show you that some favored products are, in fact, total crap.

By using nonstandard stats, the author can make anything look good... "This rifle may not be able to hit the broad side of a barn, BUT IT HAS SUCH-AND-SUCH A COOL THING! "

Then again, i'm in an especially cynically mood today, so take that into consideration :)
 
"There are three types of lies: lies, damned lies, and statistics." Mark Twain.

You can make a statistic say anything you want w/ a little "creative calculation." The truth's a little harder to fudge.

Usually AR's pretty good, but those tables were gobbledy-gook to me too. (But then, I would have preferred something like "Hevi-Shot=GOOD, Steel=BAD." :p )
 
Mike,

My apologies for not holding American Rifleman more clearly apart from the rest. I recognize and appreciate the difference. Note, though, that AR generally tests only one sample of a given gun and rarely makes note of whether the sample was provided by the manufacturer or acquired at random on the market. I have no recollection of an ammo test stating that it included mutiple lot numbers, multiple random samples of multiple models of guns, multiple shooters, etc. Even given the data AR provides, the tests are not statistically valid, and only slightly better than anecdotal. Are they superior to G&A, Shooting Times and the rest? I am sure they are.

For what it's worth, the conversation about reviews bought and paid for took place many years ago. The rep was in an environment where I could well have been mistaken for an insider and the rep had taken full advantage of an open bar for many hours. I do not recall the man's name and so cannot say how long he may have lasted at Petersen's or in the industry. Subsequent conversations with diving gear designers and manufacturers, however, only reenforced the man's assessment of his (then) employer and, to an extent, the entire publishing racket, but I certainly wouldn't tar the NRA with the same brush.
 
Why don't gun rags use standard statistics?

Because (mostly) they aren't attempting to report on the performance of the products they review. They are just whoring themselves out for whoever gives them the most ad dollars. Your mistake is thinking they are actually trying to provide useful, objective information, and just don't know how. In fact, by and large they don't care in the slightest. This is probably because gun magazines are generally such piss-poor products that they can't make money on their own merits like, say, Car and Driver does... which of course takes ads, but is more than happy to make fun of the same Chevy Silverado advertised on page 2.
 
Last edited:
They don't sell any more magazines or products for their advertisers. Only obsessed wanks pay any attention to them, and the author probably just grabs them from some other source and "bends to shape and beats to fit" as needed. (The quote is a paraphrase from product instructions in a Bugs Bunny cartoon....)
 
Okay, how many people on this board know what ANOVA is much less how to figure it?

I've had undergraduate and graduate statistics classes. Still, I'd have to blow some dust off of my statistics books to do it myself. Points are well taken that gun rags seldom bite the hand that feeds them, but I still contend that the reason higher level statistics aren't used is because gun rag writers and publishers don't understand them. If they did they could still use statistics to lie (lies, damned lies, and statistics) by choosing what to present, what not to present, and how to present it.

I use statistics in my job and I'm often surprised about the level of misunderstanding of statistical things by people that I find to be otherwise educated and sophisticated. Gun rags don't use good statistics because they simply don't get it. And, if they did most readers wouldn't get it either.

RJ
 
There's an even clearer example of misleading use of figures and standards. Look at the ranges that guns are test-fired at. With handguns, they shoot at ranges of 10, 15 or 25 yards. Why? If you're going to measure the gun against a standard, use 25 yards and be done with it. When they shoot anything larger than a pocket pistol at 15 yards they are merely doing so because it doesn't deliver credible accuracy at the longer range and they don't want to embarrass an advertiser.

Look at rifles and the 100 yard "standard". As far as I'm concerned 100 yards tells you nothing about a rifle. Any modern scoped bolt action rifle in a credible hunting caliber should be tested at its MPB range - 250 yards, say - and done so from a FIELD POSITION, not a bench rest. That's where the bad trigger, the poor ergonomics or the burr in the crown are going to tell you that you have a problem. Shooting a group off sandbags at 100 yards tells you nothing - the worst reject off the factory floor will deliver credible results from a sandbag at those ranges.

Keith
 
Some miscellaneous thoughts on sundry foregoing…

O, boy, someone's really showing off by introducing analysis of variance between groups to an on-line firearms forum! :)
I imagine most gun rag writers were either out shootin' or sittin' in jounalism or composition class while some of us were trying to wade through statistics.

Your point makes lots of sense to me but I'd guess your talking about a skill set most gun mag writers don't possess?
To repeat a recent observation elsewhere on THR, I wonder about the need of those who buy/subscribe to/read the gunzines to diminish them by referring to them with the pejorative "rags." Is it that the "gun rags" are a guilty pleasure, and so many who like to pose as critical thinkers, simply aren't?

Because if they were, they would've long ago noticed that the quintessential gunzine article follows the core formula of: "We got this gun; it looks good, shoots great and this is where you can get more product information: Excellent Gunmaker, Box 3006 - Dept. [insert magazine code], Gun Valley, USA."

Has anyone here who has written for one of the non-NRA Publications gunzines ever seen an "Editorial Guidelines" contributors sheet speaking to the issue of statistical presentation? The only one I ever saw… and I've had bylines in all of'em except the ones in Peoria since purchased by the same outfit as the former Petersen's group… was the mercifully short-lived Modern Gun whose maniac editor was more interested in photos of watermellons bursting, and five pieces of brass in the air at once. He had a six page (and ever-expanding) Contributors' Guidelines which was virtually silent on presentation of statistical data. What it did focus on, aside from the editor's pathetic ego requirements, was "the target readership," one for which the disdain was implicit.
…from my time at American Rifleman, I can tell you that for the most part fiscal considerations do NOT drive reviews.
Don't know when you were in and out of there, Mike, but the very first thing that "Red" Bell did when he was brought in from his 33 years with Petersen's, was re-make The American Rifleman in G&A's image, from the "look" of the book to the "ultra-advertser-friendly" content, the latter being a long-time Petersen's hallmark!
…AR generally tests only one sample of a given gun and rarely makes note of whether the sample was provided by the manufacturer or acquired at random on the market. I have no recollection of an ammo test stating that it included mutiple lot numbers…
You can take it as an article of faith, Huck, that aside from the editorially inconsistent Gun Tests, virtually all of the firearms tested are manufacturer or importer-provided T&E ordnance.

As for the ammo testing, I used to do that all the time for the Combat Handguns books… it was my specialty, I guess, and one I enjoyed to the utmost. I don't think anyone else ever wrang out a handgun (and the occasional rifle) with as many different loads as I did, sometimes as many as 30 distinct loadings from nine or ten manufacturers. (It really didn't mean all that much, but my editor felt that "men like to read charts," and I got paid to shoot great quantities of ammo.)

And both Frank James and to an extent, Charlie Petty in his non-TAR reports, will do the same thing with ammo, albeit to a lesser degree… except when Farmer Frank would do one of his "10,000 round melt-down" tests (that once actually melted a pistol down!) I think even TAR generally uses a maximum of three separate loadings with any firearms report.

But blades67 probably comes closest to the real reason why gunzines don't use standard statistics:
They don't know what they are.
Such a standardization would require, for starters, a symposium of Editors, and that simply isn't going to happen! While I almost never agreed with anything the late "CastleBravo" used to aver, I'll second Sean Smith's rather dour observation:
Your mistake is thinking they are actually trying to provide useful, objective information, and just don't know how. In fact, by and large they don't care in the slightest. This is probably because gun magazines are generally such piss-poor products that they can't make money on their own merits…
And they aren't gonna get any better as long as the readership pool are so woefully lacking in critical thinking skills. When enough gunzine readers write to Editors and Publishers on something other than lined notebook sheets and with a crayon, complaining at the quality of the content, even someone like Stanley Harris, who once said "Articles? Aren't they what we try to squeeze in between the advertisements?," may have to take notice.
 
Ah, Dean Speir. It has been awhile since I have seen one of your no-holds-barred commentaries on a thread. In spite of your tendency to see the worst in eyeryone, your last paragraph is dead on. Advertisers are not bashful about voicing their complaints and readers should not be either. Editors for the most part are just people trying to do a job as best they can. It would be nice if their bosses would insulate them from the marketing side and let them concentrate solely on editorial content, but sadly this is not the case. Editors bear the full brunt of advertiser's ire and only when more pressure is applied by the readership will a fairer balance be acheived. Writers, of course, also bear a large measure of responsibility.

So the next time one of the members here sees some pure hogwash in a magazine, fire up the keyboard and let them know exactly why it is hogwash.

Dan Johnson
 
Dean,

Couldn't you condense much of what you said to "Thats capitalism for you'"?
I mean, the economics of magazine sales depend more heavily on the advertisers than on the accuracy of reporting. And of course the more sales, the more you can charge your advertisers which explains the lowest common denominator aspect of many gun magazines - exploding pumpkins sell to a wider audience than accurate tables and consistent standards.

There are notable exceptions out there among both writers and magazines, so I don't want to paint with too broad a brush, but I think people should take most of what they read with a grain of salt. Look at a magazine like Gun Tests (which doesn't rely on advertisers), and you find even they fail to use any sort of consistent standard from test to test - if you could rely on the test of a single example to give you a valid assessment, which I don't think you can.

And the quality of the writing...

Can you imagine a writer for an auto magazine falling back on some of the hackneyed phrases and excuses that gun writers use? "Our Dodge Gazelle broke down in traffic six times on our first test drive, which is perfectly normal for the new-car 500 mile 'breaking in period'.... and since it's a Compact, we only drove it at a top speed of 35 mph, where we got 'acceptable handling'... The drivers wheel had quite a bit of slack and required all of our strength to turn, but that is easily remedied by any competent Carsmith... Occasionally the car would veer sharply to the left upon acceleration, exhibiting the well known one-in-five 'flyer' phenomena..."

Give me a break!

Come on, there are some very legitimate gripes associated with the industry! That doesn't mean there aren't some excellent people as well, surely you and Charley have no reason to hang you heads, and doubtless we could winnow out another dozen or two names in short order. In many ways, it seems like the more obscure the magazine, the higher the standards - which brings us full circle, back to my first paragraph.

'Nuff said.

Keith
 
Hi, Dan… glad to see you survived your "reverse migration" Eastward. Happy Noo Year from Noo Yawk!
It has been awhile since I have seen one of your no-holds-barred commentaries on a thread. In spite of your tendency to see the worst in eyeryone…
Please, if you have issue with a specific of my commentary, elaborate… while I am no longer active in any of the gunzines, if there's a new attitude behind the scenes of any of them, I'd be delighted to learn if there were any enlightened Editorial attitudes at work, or a clearly enunciated Publisher's directive to an advertising division that there is to be no interference with any Editorial perogatives.

Yes, I have the jaundiced view of a one-time "insider," but I like to think I'd educable, so please illuminate me. (I'm not being smug… I'd genuinely like to know.)

And one thing I've been processing since my inital post in this thread, it occurs to me that if there is anything that's going to, as Frank James terms it, "improve the breed" (i.e., gunzine content), I suspect that it will be the proliferation of 'Net-based fora such as the late TFL, its successor THR, AR-15.com and Cruffler.com, etc., where unredacted information is more freely available. Sooner or later, no Editor would dare allow one of his contributors to use the term "acceptable combat accuracy," and charts turned in for anything other than a Seecamp 380 or a Keltec-32 with target distances of "21 feet, the average distance for a gunfight," would result in the author being frog-marched out the door! What a wonderful thing that would be!

Of course, there are those firearms forums where no critical thinking exists at all… no names or URLs required… and others which are nothing more than the extension of the Administrator's ego or marketing requirements, but few of them even begin to approach the membership of TFL or, shortly, THR.

My ol' pal Kodiak Keith Rogan will attest to some of the ground rules established nine years ago on our comparitively little Shooting Sports BB on the old Prodigy service… starting with no discussion of "handgun stopping power" and the requirement that if one was going to discuss a certain popular brand of defensive handgun ammo, the poster had to at least know how to spell "Hydra-Shok." Charlie and I like to think we turned out a decent group of critically thinking shooters from that group. (Hiya, Keith! When are ya gonna tell everyone how best to stop a rampaging bear sow? :))
 
No, Dean, I have no specific issue with your post here. I was referring to your outlook in general having read many of your posts in the past. This is not to say I don't enjoy reading them. I do, and confess I miss your presense over at that other place.

Dan Johnson
 
(Hiya, Keith! When are ya gonna tell everyone how best to stop a rampaging bear sow? )

:what: Ouch!

Literally!!! ;)

Welcome Dean & Dan... now boys... play nice. Hehehe

starting with no discussion of "handgun stopping power"

What? No Stoppingb Power debates? OH, I know why, its cause y'all figured out that the Black Talon was the best, with Rino ammo comming in a close second. Either one of those, even loaded in... say the old .45 Long Colt, would knock someone off their feet. And if you had a clip full of either you'd be good to go for anything! :evil:
 
"Note, though, that AR generally tests only one sample of a given gun and rarely makes note of whether
the sample was provided by the manufacturer or acquired at random on the market."

Which is NO different than virtually any other magazine.

Guns ain't cheap. Consumer Reports generally only tests one of each item. Gun Tests tests only one of each item.

It's also understood that ALL of the samples are provided by the manufacturers or distributors -- NONE of AR's firearms tests are ones that were purchased on the open market. There's simply no budget for that.

As for statistical validity, American Rifleman is a MAGAZINE. It's NOT H.P. White laboratories. If you feel that you need to have a regressive analysis of how Federal XMT40SX from lots XXX, YYY, and ZZZ works in your gun, get out and get some of it and shoot it yourself.

That way you get information that NO gun magazine will ever be able to provide to you -- how a certain ammunition works in YOUR gun under YOUR shooting conditions.

I can't even imagine why anyone would mistake American Rifleman, G&A, or some of the other magazines out their for scientific journals. They're review and overview, and ATTEMPT to give anecdotal sp? (that's what it really is, folks, anecdotal) data that will be of so use to the most people.

It's not meant to be scientific. It's not meant to be indepth. When your sample is 1 gun, it can't be either.

Also, there's no staff of statisticians to do this at AR or any of the other magazines. There's no huge staff of testers (when I was there 4 people conducted all of the testing, and 1 only did a very small amount).

What I think everyone needs is to be realistic about just how much this type of data will help them. For the average shooter, I don't think it will be very much at all.

Probably the most statistically developed information that comes out of the firearms industry is from the loading manual people.

It takes years to develop a loading manual -- hundreds of thousands of man hours. And yet, even that information is kind of thin, because the loads given are normally developed in a single firemarm -- a universal receiver, a certain make of handgun or rifle, or, just perhaps, sometimes a combination of the UR and a firearm.

Come on, folks. Let's be realistic in both our expectations -- both of what we're getting from the gun rags and from what we can actually make use of in our firearms hobby.
 
Dean,

"Don't know when you were in and out of there, Mike, but the very first thing that "Red" Bell did when he was brought in from his 33 years with Petersen's, was re-make The American Rifleman in G&A's image, from the "look" of the book to the "ultra-advertser-friendly" content, the latter being a long-time Petersen's hallmark!"

Gee, that sounds vaguely familiar... :)

I was there from October 9, 1990, to April 20, 1994.

I was on Bell's staff for approximately 6 months before about 1/3rd of the Publications Staff was laid off.

Red Bell and the board cabal that hired him did more damage to American Rifleman magazine than any other single idividual ever could have hoped to do.

The faction on the board at that time (including the most useless EVP NRA has ever had) that forced his hiring wanted a G&A clone for any number of reasons long before Bell and Petersen's parted company under a cloud.

George Martin, Bill Parkerson and Tom Fulgham dragged their feet for as long as they could.

When Martin was forced out, er... excuse me, moved laterally, the writing on the wall became clear, and Parkerson bailed to go to ILA. He apparently didn't want to be around for the bloodbath that was to come. Can't blame him. At that point he had had 25 years on Rifleman and was considered to be the next director of NRA Publications and Publisher of the magazines.

Sorry, Bill. Thanks for the years of service, you :cuss: ing dinosaur, get the hell out.

Ron Keysor, Bob Hunnicutt, and Pete Dickey were all in line for the job of editor of the magazine and yet it becamse quickly apparent that it was going to be someone from the outside who would hold BOTH positions (good way to save money, I guess).

I still remember the warnings I got from a number of respected freelancers and members of the G&A staff who had worked with "good old Red." Not exactly complimentary assessments of the new boss.

Around the same time LaPierre had the "initiative" going to make the magazines self-sustaining based on their own advertising revenue, and a possible move to put Rifleman and the other NRA magazines on the newsstand.

Only problem was, with ad rates running approximate HALF of what they were for other magazines in the industry, the math isn't hard to figure out.

You can't support a 4-C 120-page book when you're charging $25,000 per page and everyone else in the industry is charging $50,000 a page and barely breaking even. The magazines had been a line item in the general budget for a long time, and apparently that rankled some people for some reason. (Interestingly enough, many of the same people were responsible for the final break between the Olympic Shooting Team and NRA.)

Bell came in, alienated the entire staff, cut 1/3rd, and lost Pete Dickey to retirement (he had no intention of doing so, until), had Ron Keysor leave for ILA because he hated working for Bell so much, had Bob Hunnicutt bail to go to work for Crosby for awhile just to get away from the BS that was going on at NRA, etc.

That done, more money needed to be saved, so usher in the age of a return to black and white photography dominating in the magazine for about a year, 10 issues a year instead of 12, a reduction of the book by almost 1/3rd, and, my favorite change of all? A virtual deemphasis on firearms content to allow Marion Hammer and other board members space for political self-agrandizement articles.

Even after all that, "good old Red, silt of the earth," still needed to save money, so when the publications groups started hiring people back to replace the people who had either been laid off or quit in disgust, guess what happened then? Starting salaries were cut for most positions, sometimes by as much as 1/3rd.

I started in October 1990 as Associate Editor for $27,000 a year. When an Associate Editor was finally rehired around 1997 or 1998, I heard through the grapevine that the starting salary was barely over $20,000.

Yep, a staff of youngsters, without experience.

Certainly a wonderful legacy for NRA and the magazines.

Now, though, those who were able to hold on through those years are in charge, and interestingly enough, they're helping bring Rifleman and Hunter back to something of the look that the magazines had before the G&Aization. Not the whole way, and to be perfectly honest, there are somethings that are very good about the old G&A style (but the last thing that's needed is a bunch of cookie cutter magazines that look exactly the same).

Mark Keefe and John Zent are people I worked with in the pre-Bell times at Rifleman. Scott Mayer, too, as well as the deputy director of Publications, Loureds Fleckenstine (sp?). Never could spell her name.

Probably most importantly, though, to a balancing of the image is the fact that Harry Jaecks is still art director. How he managed to tough it out through the Bell disaster is completely beyond me.
 
Status
Not open for further replies.
Back
Top