alan
Member
June 5, 2003
TIA Redux: Still Bad Math
by Charles V. Peña
Charles V. Peña is director of defense policy studies at the Cato Institute.
Total Information Awareness was a program built to run a massive database to find would-be terrorists. Proving that government programs never go away, the TIA is back, this time as the Terrorist Information Awareness program. It's not even the same old program dressed up in a new name. It's TIA as TIA. But if it was a bad program before, it's a worse one now.
The Defense Advanced Research Project Agency (DARPA), which launched TIA, claims that the new TIA is designed "to protect U.S. citizens by detecting and defeating foreign terrorist threats before an attack." But the core of the new TIA is the same as the old: a database of public and private records to be analyzed for patterns indicative of terrorist activities.
TIA essentially depends on the "law of large numbers. It's what marketing companies call "data mining." In a nutshell, "profiles" are developed of people who should be good customers for a particular product or service. A large pool of people who fit the profile is targeted, knowing that only a small fraction will actually be customers. That's what TIA will do. Only instead of potential customers, the profiles will be for would-be terrorists. And like commercial data mining, only a small fraction of the pool of people who fit the profile of a terrorist will, in fact, be actual terrorists.
So how hard or bad can that be? Well, a little "back of the envelope" analysis helps shed some light on TIA. Assume a population of 240 million adults (i.e., children are not would-be terrorist candidates). Assume there are 5,000 terrorists lurking among us. Assume a 99.9% probability (i.e., near perfect and very highly unlikely) of correctly identifying a suspect as an actual terrorist -- that is, if you suspect someone is a terrorist, he is actually a terrorist. And assume a 99.9% probability (again, highly unlikely) of correctly identifying a suspect as an innocent person.
What's the probability of finding a terrorist hiding among the masses? Without boring you with all the math involved, here's the answer:
244,299 people will be identified as suspected terrorists (and remember that this is with near-perfect accuracy of being able to correctly identify terrorists and innocent people).
239,995 innocent people will be mis-identified as terrorists.
The probability of finding a real terrorist is 2 percent.
The inescapable conclusion is that TIA, if broadly applied, is simply a fishing expedition that casts a wide net, snaring many to catch very few.
What happens if the accuracies for being able to correctly predict whether someone is a terrorist or innocent person are reduced from 99.9% to 90% (still very high and unlikely)?
Over 24 million people will be identified as suspected terrorists.
Slightly less than 24 million innocent people will be mis-identified as terrorists.
The probability of finding a real terrorist is .02 percent or nearly zero.
That certainly doesn't bode well for the prospects of TIA.
But what if we reduce the total pool of potential terrorists to just the U.S. Muslim population (not an entirely unreasonable assumption given that al Qaeda is a radical Islamic terrorist organization) or about 6 million people and we split the difference on accuracy to 95% (which is still very high and not a "real world" statistical norm)?
304,500 people will be identified as suspected terrorists.
299,750 innocent people will be mis-identified as terrorists.
The probability of finding a real terrorist is 1.5 percent.
Still not very good, is it?
Here's one final example. Assume we're looking for 19 hijackers within the U.S. male Muslim population, about 3.6 million people. And assume near-perfect 99.9 percent accuracy. How hard would it have been to find the hijackers and potentially avert 9/11?
3,619 people would have been identified as suspected terrorists.
3,600 innocent people would have been mis-identified as terrorists.
The probability of finding a real hijacker is about one-half of 1 percent.
So when it comes to TIA, you do the math.
And here's one last cautionary note about TIA. Advocates are claiming that it is a solution to "connecting the dots" of intelligence data that weren't connected prior to 9/11. And, to be fair, if the pool of people the TIA methodology is applied to is small enough and focused (i.e., you already have good reason to suspect someone might be a terrorist), it might be a useful tool. But another major intelligence shortcoming was over-reliance on technical intelligence gathering and not enough "old fashioned" human intelligence. Revisiting TIA will be going "back to the future" and falling into the same trap.
1000 Massachusetts Avenue, N.W. Washington D.C. 20001-5403
Phone (202) 842-0200 Fax (202) 842-3490
All Rights Reserved © 2003 Cato Institute
TIA Redux: Still Bad Math
by Charles V. Peña
Charles V. Peña is director of defense policy studies at the Cato Institute.
Total Information Awareness was a program built to run a massive database to find would-be terrorists. Proving that government programs never go away, the TIA is back, this time as the Terrorist Information Awareness program. It's not even the same old program dressed up in a new name. It's TIA as TIA. But if it was a bad program before, it's a worse one now.
The Defense Advanced Research Project Agency (DARPA), which launched TIA, claims that the new TIA is designed "to protect U.S. citizens by detecting and defeating foreign terrorist threats before an attack." But the core of the new TIA is the same as the old: a database of public and private records to be analyzed for patterns indicative of terrorist activities.
TIA essentially depends on the "law of large numbers. It's what marketing companies call "data mining." In a nutshell, "profiles" are developed of people who should be good customers for a particular product or service. A large pool of people who fit the profile is targeted, knowing that only a small fraction will actually be customers. That's what TIA will do. Only instead of potential customers, the profiles will be for would-be terrorists. And like commercial data mining, only a small fraction of the pool of people who fit the profile of a terrorist will, in fact, be actual terrorists.
So how hard or bad can that be? Well, a little "back of the envelope" analysis helps shed some light on TIA. Assume a population of 240 million adults (i.e., children are not would-be terrorist candidates). Assume there are 5,000 terrorists lurking among us. Assume a 99.9% probability (i.e., near perfect and very highly unlikely) of correctly identifying a suspect as an actual terrorist -- that is, if you suspect someone is a terrorist, he is actually a terrorist. And assume a 99.9% probability (again, highly unlikely) of correctly identifying a suspect as an innocent person.
What's the probability of finding a terrorist hiding among the masses? Without boring you with all the math involved, here's the answer:
244,299 people will be identified as suspected terrorists (and remember that this is with near-perfect accuracy of being able to correctly identify terrorists and innocent people).
239,995 innocent people will be mis-identified as terrorists.
The probability of finding a real terrorist is 2 percent.
The inescapable conclusion is that TIA, if broadly applied, is simply a fishing expedition that casts a wide net, snaring many to catch very few.
What happens if the accuracies for being able to correctly predict whether someone is a terrorist or innocent person are reduced from 99.9% to 90% (still very high and unlikely)?
Over 24 million people will be identified as suspected terrorists.
Slightly less than 24 million innocent people will be mis-identified as terrorists.
The probability of finding a real terrorist is .02 percent or nearly zero.
That certainly doesn't bode well for the prospects of TIA.
But what if we reduce the total pool of potential terrorists to just the U.S. Muslim population (not an entirely unreasonable assumption given that al Qaeda is a radical Islamic terrorist organization) or about 6 million people and we split the difference on accuracy to 95% (which is still very high and not a "real world" statistical norm)?
304,500 people will be identified as suspected terrorists.
299,750 innocent people will be mis-identified as terrorists.
The probability of finding a real terrorist is 1.5 percent.
Still not very good, is it?
Here's one final example. Assume we're looking for 19 hijackers within the U.S. male Muslim population, about 3.6 million people. And assume near-perfect 99.9 percent accuracy. How hard would it have been to find the hijackers and potentially avert 9/11?
3,619 people would have been identified as suspected terrorists.
3,600 innocent people would have been mis-identified as terrorists.
The probability of finding a real hijacker is about one-half of 1 percent.
So when it comes to TIA, you do the math.
And here's one last cautionary note about TIA. Advocates are claiming that it is a solution to "connecting the dots" of intelligence data that weren't connected prior to 9/11. And, to be fair, if the pool of people the TIA methodology is applied to is small enough and focused (i.e., you already have good reason to suspect someone might be a terrorist), it might be a useful tool. But another major intelligence shortcoming was over-reliance on technical intelligence gathering and not enough "old fashioned" human intelligence. Revisiting TIA will be going "back to the future" and falling into the same trap.
1000 Massachusetts Avenue, N.W. Washington D.C. 20001-5403
Phone (202) 842-0200 Fax (202) 842-3490
All Rights Reserved © 2003 Cato Institute