By James Careless
Redundancy is at the heart of California’s statewide upgrade to Next Generation 911 (NG911) services. This is why, when it allocated nearly $1 billion to this project in 2019, the State decided that selecting four different vendors to complete and run the system in concert with the California Office of Emergency Services (CalEOS) was a wiser choice than selecting a single vendor solution. These vendors are Atos, Lumen (formerly CenturyLink), NGA and Synergem Technologies.
“When we began the process, there was a state to the north of us that had a statewide outage because they had a single provider for their entire state,” said Budge Currier, Assistant Director of Public Safety Communications. “We did not want that to happen to us, given the size of our state.”
Of course, choosing four vendors to serve the state rather than just one brings its own challenges, namely ensuring seamless interoperability between their respective equipment and software sets. To make this happen, CalEOS established a special laboratory to test, tweak, and coordinate interoperability on a statewide scale. It also had to ensure that the redundant ‘failover’ procedures allowing one vendor to cover for another during an outage function as planned.
Fortunately, a state the size of California has the financial resources to fund such an interoperability project and cope with the time delays associated with such an effort. “All of our funding comes from the State Emergency Telephone Number Account (SETNA),” said Currier. That’s an option that could be out of reach for less prosperous jurisdictions.
A Very Big Project
The goal of the California NG911 project, as guided by its “NG9-1-1 Transition/Implementation Plan”, is to upgrade the state’s PSAPs (Public Answering Safety Points) to support all NG911 services. “NG911 will allow 911 centers to accept and process a range of information from responders and the public, including text, images, video, and voice calls,” said cisa.gov, the website of the US government’s Cybersecurity & Infrastructure Security Agency. These are capabilities far beyond what can be offered by legacy 911 systems.
To put it mildly, California’s NG911 Project is big — very big. “There are 450 PSAPs on the solution and there are 26.6 million 911 calls a year that will run through the solution,” said Currier. “We have built out an IPv6 network that has over 4,000 endpoints in the network and we’ve integrated a private key infrastructure into that network [as well].”
“We’re being told that this is one of the larger IPv6 networks in the world, which we thought was kind of amazing, but it is true,” he added. “The scale and scope of this is huge. It’ll certainly be the largest system of its kind that anyone builds, mainly because other states that are similar or slightly smaller than us are going to go with a more regionalized approach instead of a statewide approach.”
For the record, California’s NG911 system software is built upon the NG911 i3 standard developed by the National Emergency Number Association (NENA). In turn, the i3 standard is founded upon “the concept of an ESInet (Emergency Services IP Network), which was designed as an inter-network that could be shared by all 911 agencies involved in an emergency,” explained www.nga911.com, the website of vendor NGA. Adopting this standard means that all California PSAPs “would be capable of receiving IP-based signals and multimedia delivery of emergency information.”
Why California Is Pursuing NG911 So Vigorously
Paying nearly $1 billion to upgrade to NG911 is a massive investment for any state, even one as large as California. But it is an investment that has to be made, due to shortcomings with their legacy 911 system. It’s not just a matter of being unable to work with IP-based signals and multimedia delivery. “When we first started this journey, and this is public information, we were seeing about 17,000 minutes of outage per month with the legacy system,” said Currier. “That means that some PSAP somewhere in California was not able to receive 911 calls because the 911 system itself was down.”
Since 2019, California has seen this legacy 911 outage rate grow. “Now that number is anywhere between 50,000 and a hundred thousand minutes a month,” he said.
Another reason California is pushing hard for NG911 is the legacy 911 system’s inability to easily move 911 calls from one PSAP to another. Although the existing system does allow calls from one PSAP to be rerouted to the next closest PSAP during an emergency, it can’t do much more than that. In contrast, the state’s NG911 system will support redundant options for 911 call rerouting, as we will see later on in this article.
This shortcoming in California’s legacy 911 system made itself apparent during the February 2, 2017 failure of the 770′ Oroville Dam, the largest of its kind in the United States. “We had six PSAPs evacuate at the same time,” said Currier. Since the existing system only associated each PSAP with a single backup PSAP to reroute its calls to, losing six at the same time left 911 callers in the Oroville, California region with “nowhere for those 911 calls to go,” he noted. “So we wanted to remove that challenge from the network.”
This state is also committed to NG911 because this new technology supports improved call routing — in other words, sending 911 calls and messages to the nearest PSAP to the caller, so that the nearest first responders can come to their aid. “That was the main motivation for moving to NextGen 911,” said Currier. “And now that we are beginning the deployment of these systems), we are actually seeing huge gains in every single one of those areas.”
Redundancy By Design
California’s NG911 plan could be described as ‘redundancy by design’. The state is broken into four NG911 regions provisioned by three vendors, with the entire state having access to a NG911 network provisioned by a fourth vendor as well.
The northern California region — “the San Francisco Bay area over to Sacramento, north up to Oregon,” Currier said — is handled by Synergem Technologies. The Central region runs south from there to Los Angeles County, which has been set aside as a third separate NG911 region. These two regions belong to NGA, while the Southern region is Lumen’s territory. Finally, Atos has been selected as California’s statewide NG911, with connections to all of its PSAPs.
When it comes to providing NG911 signal redundancy, “we’ve got failovers at every single step of the network,” said Currier. “If there’s a failure anywhere downstream, the region can fail over to the statewide provider. The next piece is the NG911 core: Each region has two different cores that can fail over one to another within their own solution before failing over to the region-wide provider. And then the final piece is the (ESInet) network that connects the NextGen 911 routing and core services to the PSAP. It can fail over as well. So we’ve built in that redundancy every single step of the way, and we’ve tested all of that throughout the entire ecosystem.”
The Vendors’ Contributions
Each of California’s NG911 vendors are contributing something vital to this project.
In its role as California’s statewide NG911, Atos is serving as “the primary network service provider (PNSP)”, said Joe Hernandez, Head of Attos Public Safety in North America. “As the PNSP, we are providing NG911 core services throughout the entire state, serving the entire population and capable of routing all 911 there. Now we do operate with and in collaboration with the three regional network service providers (NGA, Lumen, and Synergem Technologies). So we interconnect with them and work as their backup in many cases as well,”
In addition, Atos is California’s primary provider of text-to-911 services, Hernandez said. “This comes in two flavors. One flavor is what we call ‘over the top’, which uses a browser for those PSAP systems that don’t integrate into the i3 network. And then we have the fully integrated version of text-to-911 for those PSAPs that are fully i3 compliant end to end. We also handle the geospatial routing for text-to-911 throughout the state and can serve as a secondary NG911 voice provider should one of the regional providers have outages, among other things. We’re here to keep 911 resilient.”
As for Lumen? “We provide an emergency service IP network, IP-based software services and applications as well as core call routing services,” said Jim Carlson, Lumen’s Senior Director of Public Safety. “Our Lumen NG911 platform leverages NENA i3 standards-based technology that integrates with the NG911 state-wide provider.”
NGA 911 had made a number of contributions to this project. In addition to being assigned the Central and Los Angeles County regions, “NGA was awarded another piece of the NG911 rollout as a Cloud native Call Handling System (CHS) provider,” said Barb Winn, the company’s Chief Growth Officer. “In 2022 NGA was the first CHS provider to be approved out of the CalOES lab and authorized to sell CHS in CA. Now in 2023, NGA is the first NG911 provider to go live in California with a Cloud native CHS at the Desert Hot Springs Police Department, and many other CHS roll-outs in progress.”
“The equipment we deploy for NGCS (NG911 Core Services) and ESInet takes minimal rack space in a PSAP’s backroom,” added Rebecca Dungey, NGA Marketing Director. “NGA is also offering CHS throughout the entire State of California. Our CHS deployments provide everything the PSAP needs to handle 911 calls without the large footprint of legacy Customer Premises Equipment.”
For its part, Synergem Technologies has already built a nationwide i3 network that provides a full range of NG911 services. “This allows us to offer these services to jurisdictions throughout the country,” said company COO Jeff Schlueter. “That’s how we got into California. We are already provisioning NG911 services in Wisconsin, upper Michigan, Florida, and Washington — and now going into New Jersey in partnership with Lumen.”
Making Multiple Vendors Interoperable
There’s an old saying that goes, ‘when you have one clock, you know the time. When you have two clocks, you never know the time.”
This saying alludes to the challenges that occur whenever an organization has two or more devices performing the same function, or in the case of California’s NG911 system, four vendors integrating their equipment into the same network. Granted, their mutual use of the NENA i3 standard ensured that there were no fundamental incompatibility issues that could not be resolved. Still, there are usually subtle differences between ‘shared standard’ equipment made by different vendors that can potentially cause unforeseen problems during and after deployment — unless they are identified prior to deployment and addressed.
This possibility was not lost on the California Office of Emergency Services. “We knew that would be a challenge,” said Currier. “This is why we brought all the vendors into the CalOES lab to make sure that they could operate with one another.”
In general, CalEOS and its vendors adhered to the i3 standard as closely as they could. In those areas where it provided options for them to choose from, “we would come to a consensus and say, ‘this is the way we’re going to do it,’” he said. “For instance, we developed an interface control document that outlines how each of them will handle a call flow and then we just got them in there and tested it.”
Now hammering out solutions between CalOES and its NG911 vendors can be a time-consuming process. Still, it is vastly preferable to work out issues at this stage of the project than to encounter them after NG911 has gone live in California.
“It was very, very challenging to work through that process, but it’s been successful and we now have a multi-vendor solution that fails over and complies with the i3 standard,” said Currier. “And this is one of the first deployments — and arguably maybe the first complete standards-based deployment — using the NENA i3 standard. So we’ve been able to go back to the NENA standards body and make some recommendations on how to actually improve the standard as a result of the work that we’re doing here.”
In addition to this effort, Atos has been coordinating with California’s three regional NG911 providers to keep the project running smoothly. “We conduct at least a dozen coordination meetings every week with CalOES, the three regional service providers, and the entire NG911 supply chain,” Hernandez said. “We also have very sophisticated progress measurement systems to track how we’re doing on a PSAP by PSAP basis, which we share with all of the players in this project.”
Lumen’s Jim Carlson is a big supporter of this collaborative approach. “As with any complex project, coordination between CalOES and the vendor community requires regular communication,” he said. This is why “For example, CalOES hosts face-to-face sessions to discuss project status, communication and deliverables.”
Working together is paying off, said Winn. In fact, “NGA is grateful to our partners in delivering NG911 throughout the State of California for the progress made thus far,” she said. “Working groups and the CalOES lab have greatly helped with information sharing, Interoperability Testing, and how to interpret i3 and other industry standards.”
Dealing with Delays
Originally, California had hoped to complete rolling out NG911 statewide by the end of 2022. But as often happens in projects this big and complex, deadlines have slid.
One reason is the work involved in ensuring i3 interoperability between all four vendors. In this area, “legacy technology and process has been the most difficult challenge to overcome,” Winn said. “Any implementation this large is a challenge.”
The pandemic also messed things up by slowing down deployments and supply chains. “Yeah, COVID threw a bit of a wrench into the works,” said Schlueter. “We physically couldn’t get into the PSAPs where we needed to install equipment. So we used FaceTime or other video conferencing applications, and had the PSAP personnel show us around their data rooms so that we could perform site surveys.” Unfortunately, you can only see so much during a remote site survey, leading Synergem’s techs to run into unforeseen issues once they got physical access to these PSAPs later on.
No one could have planned for the unforeseen delays imposed by COVID-19. As for those caused by CalOES’ multi-vendor approach, namely the time spent on achieving interoperability between vendors at the CalOES lab? Budge Currier is prepared to live with them, given the element of redundancy this approach has given to California’s NG911 system
“I think it’s definitely justified by the redundancy,” he said. “What we didn’t really anticipate is that vendors would help one another out (to solve issues); what one vendor misses, another one finds. And so we’ve found huge advantages from the collaboration between these companies. You’re bringing a brain trust of experts together that are really designing a unique solution that we would not be able to do had we done it just with ourselves or with a single vendor.”
As for where California’s NG911 roll-out stands today? “We admittedly are about a year behind schedule; we’ve been pretty vocal about that,” Currier replied. “We have four different counties that have PSAPs in them that are active on the NG911 solution, and we have about 25 PSAPs that are either directly receiving NextGen 911 calls or are capable of having such calls transferred to them. Our goal is to be done next spring. I fully anticipate we’ll be mostly done by then and aside from those PSAPs that are moving or have unique circumstances that we need to accommodate, I think we’ll be pretty close to being done at that point.”
The Bottom Line
The applicability of California’s multi-vendor NG911 approach to other states depends on their needs, their timelines, and most of all, their budgets. A near $1 billion price tag may be too rich for many of them, and the efforts required to achieve multi-vendor interoperability may be too much to ask compared to a single vendor NG911 solution that handles it all.
This being said, California’s goal in pursuing a multi-vendor approach was to add redundancy to their NG911 system and end the system outages associated with legacy 911. On this point, Budge Currier is well satisfied with the benefits this project is accruing to his state.
“I think the biggest one is improved reliability: The system’s not going to go down anymore,” he said. “The other big one that we see, aside from the other advantages I have already listed, is our NG911 deployment becoming a centerpiece and a catalyst for discussing how technology can be used to help and improve 911 centers. And then the final one would be improved cybersecurity. Legacy 911 is not able to implement cybersecurity protections. NextGen 911 is.”
“This project is a success because California drove the roll-out, not vendors,” added Winn. “ Through our joint efforts, we have developed an NG911 approach that is truly standards-based, And in spite of challenges, we have been able to achieve interoperability.”
The bottom line: Although the near $1 billion price tag of California’s NG911 roll-out may make some eyes water, it appears you can’t argue with the results. At least, this appears to be the case, but we’ll know better after its NG911 system has been in operation for a few years or so.