NYC 911 system down; hope no one needed the police to defend them!

[papercut]

I didn't hear about this until today. I hope no one needed to use his or her phone in self defense!

http://www.ny1.com/ny/TopStories/SubTopic/index.html?topicintid=1&subtopicintid=1&contentintid=38454

Verizon: Human Error Led To 911 System Shutdown
MARCH 27TH, 2004

Verizon now says that human error led to the city's 911 system being knocked out for about two hours last night during a software upgrade.

Dispatchers say they noticed the problem around 7:30 p.m. when the volume of callers slowed down.

Verizon says calls were rerouted during a procedure to upgrade service for a corporate client.

But the company says a backup system designed to ensure the 911 system wasn't affected didn't kick in due to human error. Verizon isn't getting any more specific.

The outage affected parts of Brooklyn, Staten Island and Queens.

“Once the problem was detected, the city gave out direct numbers to police precincts, fire borough command offices and EMS,†said Information Technology Commissioner Gino Menchini. “Every effort was made to ensure the public had access to public safety officials.â€

During the outage, police say 80 calls that would have gone to 911 went to the non-emergency 311 number instead.

The 911 system was fully restored by 10 o'clock.

Meanwhile, politiicians are speaking out saying improvements need to be made to the emergency system.

Two members of the City Council's Public Safety Committee say a better backup system must be in place and that Verizon should be held accountable for its mistakes.

"This is Verizon's second major failure in less than a year," said Councilman Peter Vallone Jr. "They failed us during the blackout and they failed us last night. Last night should be an emergency wakeup call to the city: we need an answer to those questions."

"Now to know that the 911 system is that vulnerable to these routine types of disruptions is a very serious concern," said Councilman Vincent Gentile.

Verizon says it will take immediate steps to make sure this doesn't happen again.

and

http://www.nytimes.com/2004/03/28/nyregion/28emergency.html?pagewanted=print&position=

Verizon Plans Steps to Prevent Another Shutdown of 911 Line
By WINNIE HU

Verizon began taking steps yesterday to better protect New York City's 911 emergency line after a data error by an employee brought down the system in Brooklyn, Queens and Staten Island for about two hours on Friday night, city and Verizon officials said.

The emergency system broke down about 7:20 p.m. after a Verizon engineer who was making service changes to a bank's telephone numbers in Brooklyn inadvertently included numbers that are used to carry 911 calls, city and telephone company officials said. The numbers were close in sequence, the officials said.

The 911 calls then ended up being rerouted to the bank's phone system, and callers heard a busy signal. City and Verizon officials said that while the backup system in place for 911 was functioning properly, it failed to pick up the calls because it was designed to catch a technical error, not a human error that would be interpreted as simply a change of instruction.

Daniel Diaz Zapata, a Verizon spokesman, said the telephone company would now require a second person to double-check any entry of data that could affect the 911 system, and said the company planned a thorough review of its procedures that would be documented in a report to the city within a few days.

"We determined that a human error resulted in the accidental rerouting of phone calls during a procedure to upgrade service for a corporate client," Mr. Zapata said. "We have immediately altered our processes to ensure this type of situation does not reoccur. We have assured the city that we took immediate steps to make sure this doesn't happen again."

Citing privacy concerns, Mr. Zapata declined to identify the Verizon engineer, except to say that he was a veteran of the company. Mr. Zapata said it was unlikely that disciplinary action would be taken against him.

Police and fire officials said yesterday that they had no reports of injuries during the 911 failure. Fire officials said that about 60 firefighters responded to a major fire, at 3301 Foster Avenue in Brooklyn, which was called in at 8:49 p.m. by someone using a fire alarm box on the street. There were no injuries in the fire.

Paul J. Browne, the Police Department's deputy commissioner for public information, said the department immediately adopted emergency procedures, like requiring e officers on patrol to turn on their flashing lights so people could find them easily and increasing staffing at precinct station houses to answer phone calls. But he said there was no reported increase in crime.

"This didn't present an opportunity for the criminally minded - like the blackout did - because probably most people were unaware that it was out of service," he said.

However, several City Council members expressed anger that the 911 system could have been so easily disabled, and called for creating a more effective backup procedure.

"It's an emergency wakeup call," said Councilman Peter F. Vallone Jr., the chairman of the Public Safety Committee, who plans to hold a hearing about the incident. "We don't have an adequate backup system for 911, which is more important than ever as we fight the war against terrorism."

Gino P. Menchini, the commissioner of the Department of Information Technology and Telecommunications, said city officials were working with Verizon to ensure that the emergency system's numbers were clearly identified, and that its software and equipment were protected from similar human errors.

But Mr. Menchini emphasized that the emergency system already had many built-in safeguards, such as the ability to route 911 calls through either of two central offices and their 911 answering centers. "The bottom line is, 911 works very well, and it's worked very well for a long time," Mr. Menchini said.

Several emergency services experts agreed yesterday with Mr. Menchini, saying that New York 911 system compared favorably with those in other large cities and that an error like the one made by Verizon could not necessarily have been prevented because it was not a flaw in the 911 system itself.

"It's very unusual for that to happen, but it's understandable," said Robert C. Krause, executive director of Emergency Services Consultants in Toledo, Ohio, who is familiar with the New York system. "I don't know if anyone would anticipate this because it's a highly technical thing. I think most public safety administrators would assume that their numbers are safeguarded."

Mayor Michael R. Bloomberg did not have any public events scheduled yesterday, and it was unclear whether he was in the city at the time of the 911 failure. "We don't comment on the mayor's whereabouts," said Jordan Barowitz, a spokesman for the mayor. His office referred questions on the disruption to Mr. Menchini.

 

[larry_minn]

IMO nothing can be made fool proof. The dang fools are too inventive. Stuff like this WILL continue to happen. I program in a number of direct phone numbers to local cities in my cell phone as well as direct cell numbers of squad cars. I do notice more places (public) they phone just has a 911 sticker for Police/Fire/EMS. They don't even post the direct numbers so you have to find a book and look them up. (not always even posted in front of book.