Oakland's 911 outage and fallback plans that fail together

A fallback plan fails at the same moment as the primary when both depend on the same physical thing. Oakland's 911 outage last month is a good reason to check for that, and it raises a question every emergency communications center should be able to answer before it happens to them.

At about 8:30 p.m. on Wednesday, Oct. 21, Oakland police dispatchers noticed that the computer-aided dispatch (CAD) system had frozen and was showing an error. According to CBS San Francisco, dispatchers tried both emergency and non-emergency lines and found no service on any phone line, internet, or CAD. Police and AT&T later said both copper and fiber optic lines had been cut. The outage ran for more than 24 hours. CAD and internet came back Thursday night, and the city said phone service to all city systems was back by Friday morning, according to ABC7.

The line I keep coming back to is from Police Chief James Beere. The city has contingency plans for this situation, he said, but "unfortunately both of the initial fallback plans were compromised." ABC7 reported that he did not elaborate on what those plans were.

What I know and what I do not

I do not know what Oakland's fallback plans were or why they failed. The chief did not say, and I am not going to fill in that gap. Police are treating the cut lines as a criminal matter, and that investigation is theirs. This post is not about who or why.

What the public record does show is the response that worked. 911 calls were transferred to the Alameda County Sheriff's Office, and police dispatchers relocated to the sheriff's office and the Oakland Fire Department, per CBS. On Thursday, the department was responding only to "priority one calls," ABC7 reported. The chief also said he had never seen the optic lines go out and the phones disconnected in a way that also affected the fire department's system. The city says it will continue to work with AT&T to harden critical infrastructure and strengthen backup systems.

Oakland was not alone that week. ABC7 noted that Morgan Hill and Gilroy police had non-emergency lines down in outages caused by copper wire cutting, while their 911 service was not affected, and that there were outages in Riverside and Santa Barbara counties too.

So the lesson does not depend on Oakland's specifics. Any public safety answering point, hospital, or utility that wrote "if the primary fails, we switch to X" should ask a narrower question: what does X depend on, and is any of it the same as what the primary depends on?

Why fallbacks share fate

Fallbacks usually get designed as logical alternatives. A second phone trunk. A backup circuit. A cellular failover router. Transfer to a neighboring agency. On a diagram, each looks independent.

In the ground, they often are not. Two circuits from the same carrier can leave your building through the same conduit, run through the same central office, and share the same pole line for miles. A failover router may need the building's network core, which needs the same power and the same switch closet as everything else. Call transfer to a neighbor depends on the network path that carries the transfer. Even "relocate staff to the backup site" depends on that site having its own connectivity rather than a tail off the same route.

None of this shows up in a plan that lists systems by name. It shows up only when you trace each dependency down to the physical layer and look for overlap.

A method for finding shared dependencies

Here is how I would run it. It takes a few working sessions, a whiteboard, and people from IT, telecom, facilities, and operations in the same room.

1. List the critical functions, not the systems. For a dispatch center that means receiving 911 calls, receiving non-emergency calls, dispatching units, radio, and access to CAD and records.
2. For each function, write the primary path and every fallback in the order you would use them.
3. Trace each path down to things you could point at: carrier, circuit ID, building entry point, conduit, central office, power source, and the network gear in between. Ask the carrier for route documentation. If it cannot provide it, write that down as a finding.
4. Put every path on one sheet and mark each element that appears in more than one path. Those marks are single points of failure. They are often the same few things: one building entry, one carrier office, one power panel.
5. For each shared element, ask what would happen if it disappeared. If the answer is that the primary and the first fallback both stop, you have found what the chief described.
6. Fix the worst overlaps first. Typical fixes are diverse physical entry, a second carrier on a different route, a different medium such as wireless or satellite, and transfer agreements with agencies that do not share your path. Paper procedures for dispatch count as a fallback too, as long as people have practiced them.

The output is a short dependency map, and it needs an owner who updates it whenever a circuit, carrier, or building changes.

Test the map with a tabletop

A map is a claim. A tabletop is how you test whether people can act on it. I would write the inject plainly: at 8:30 p.m., CAD freezes and every phone line and internet connection at the primary center is dead. Then ask the room to walk through each fallback in order, and at each step ask what that fallback needs in order to work.

Questions I would put on the facilitator sheet:

• Who declares the outage and starts the transfer, and how do they reach the receiving agency if your own phones are down?
• How do dispatchers reach the backup location, and what do they find when they get there?
• How do you decide which calls get a response when capacity drops? Oakland moved to priority one calls. Does your plan say who makes that decision?
• How do you tell the public what to do, if the city's own systems are part of the outage?
• When do you declare service restored, and how do you confirm calls are reaching you again?

Put telecom and facilities at the table alongside dispatch leadership. The people who know where the conduits run are usually the ones who never get invited.

For a structure to start from, Cyber Defense Tactics has a free AI tabletop exercise kit with a facilitator guide, and the outage inject above fits its format.

Pick your primary 911 path and your first fallback. Can anyone in your organization tell you, today, whether they leave the building through the same conduit?