One of my former mentors, Deputy Chief (Ret.) Jim Graham of the Chesterfield County (VA) Fire and EMS Department, was a huge proponent of the use of information, whether on a fireground operation, developing a new training program, or addressing budget reductions. One of his favorite sayings—one that stays with me to this day—was:
“We must constantly strive to become better at data-driven decision making, instead of following the ‘I think, I feel, or I believe’ model.”
However, he usually followed that up with the caveat that we should strive to ensure that we had good information, not just any information.
This is an area in which emergency dispatch could improve: the ability to collect accurate, complete, and meaningful data from across the USA (or the world) about common issues or problems, including: recruitment, hiring, and retention of staff, leadership and management practices, quality assurance and improvement, and the impact of new technologies.
A data-driven success story
In 1964 and 1966, public pressure grew in the United States to increase the safety of cars, culminating with the publishing of “Unsafe at Any Speed” by activist lawyer Ralph Nader and a report from the National Academy of Sciences entitled “Accidental Death and Disability—The Neglected Disease of Modern Society.” That 37page booklet, commonly referred to as the “White Paper,” addressed the huge and costly problem of accidental deaths and injuries in the United States. In 1965, there were 52 million accidental injuries leading to 107,000 deaths, 10 million temporarily disabled persons, and 400,000 permanently impaired individuals. Those deaths and injuries cost an estimated $18 billion (about $129 billion in 2012 dollars). The White Paper stated that accidents were the leading cause of death for persons aged 1-37 and the fourth leading cause of death for all ages in 1965. For people under 75, motor vehicle accidents constituted the leading cause of accidental death.
Public opinion moved the United States Congress to enact the National Traffic and Motor Safety Act in 1966. The Act created the National Highway Safety Bureau (now the National Highway Traffic Safety Administration) and empowered the federal government to set and administer new safety standards for motor vehicles and road traffic safety. The Act was one of a number of initiatives by the government in response to the increasing number of cars and associated fatalities and injuries on the road following a period when the number of people killed on the road had increased 6-fold and the number of vehicles was up 11-fold since 1925.
So what happened? As a nation, we took a systematic approach to solving the problem by creating the modern trauma care system and the Emergency Medical Services to reduce the mortality and morbidity rates for motor vehicle crashes. We enacted new standards and regulations and requirements for automobile safety so that the automobile manufacturers were motivated to engineer and build safer cars. The automobiles they started to produce included lap/shoulder belt restraint systems, air bag restraint systems, energy-absorbing steering columns, and vehicle chassis construction that dissipates crash energy to protect vehicle occupants.
Finally, planners and engineers and road builders started to design and build safer roads for those safer vehicles to travel on. Those road construction design improvements included guard rails to prevent vehicles from striking stationary objects, e.g., bridge abutments. Guard rails, along with better banking in road curves to promote better contact between the vehicle and the road, became important tools in preventing vehicles from leaving the road on tight curves or when crossing into on-coming traffic. The reduction of the rate of death attributable to motor-vehicle crashes in the United States represents the successful public health response to a great technologic advance of the 20th century—the motorization of America.
That’s the “why” (the deaths, injuries, and costs to society were unacceptable) and the “what” (measures were taken to reduce those numbers) concerning our nation’s response to accidental death and injuries. So how did we get there? We collected data from across the nation from hospitals, state health departments, law enforcement agencies, and many other organizations that helped us understand the accidental death and injury problem. We analyzed that data and created meaningful information, information that drove the thousands of decisions that started in 1966 and ultimately had a positive impact on reducing accidental deaths and injuries from motor vehicle crashes in the United States.
Data, decision making, and judgment
Unfortunately, we don’t seem to have taken all the lessons of the White Paper to heart. Management education today is still largely about educating for judgment—developing future leaders’ pattern-matching abilities, usually via exposure to a lot of case studies and other examples, so that those leaders will be able to confidently navigate the business landscape. Whether or not they’re in business school, these leaders are told that they can trust their guts and their instincts and that once they’ve gained some experience, they can make accurate assessments in the blink of an eye.
This is a most harmful misconception. Human intuition is real, but it’s also really faulty. Human parole boards do much worse than simple formulas at determining which prisoners should be let back on the streets. Highly trained pathologists don’t do as good a job as image analysis software at diagnosing breast cancer. Purchasing professionals do worse than a straightforward algorithm at predicting which suppliers will perform well. America’s top legal scholars were outperformed by a data-driven decision rule at predicting a year’s worth of Supreme Court case votes.
When presented with this evidence, a contemporary expert’s typical response is something like, “I know how important data and analysis are. That’s why I take them into account when I make decisions.” While this sounds right, it’s completely wrong. Here the research is clear: When experts apply their judgment to the output of a datadriven algorithm or mathematical model (in other words, when they give themselves the opportunity to secondguess it), they generally do worse than the algorithm alone. As sociologist Chris Snijders puts it, “What you usually see is [that] the judgment of the aided experts is somewhere in between the model and the unaided expert. So the experts get better if you give them the model. But still the model by itself performs better.”
Things get a lot better when we flip this sequence around and have the expert provide input to the model, instead of vice versa. When experts’ subjective opinions are quantified and added to an algorithm, its quality usually goes up. So pathologists’ estimates of how advanced a cancer is could be included as an input to the image-analysis software, the forecasts of legal scholars about how Supreme Court justices will vote on an upcoming case could improve the model’s predictive ability, and so on.
Of course, this is not going to be an easy switch to make in most organizations. Most of the people making decisions today believe they’re pretty good at it, certainly better than a soulless and stripped-down algorithm. They also tend to believe that using models and algorithms takes away from their decision-making authority and reduces their power and value. The first of these two perceptions is clearly wrong, the second a lot less so.
The practical conclusion is that we should turn many of our decisions, predictions, diagnoses, and judgments—both the trivial and the consequential—over to the algorithms. There’s just no controversy any more about whether doing so will give us better results. Emergency communications managers are no different from other kinds of experts in this regard. These leaders must improve their ability to collect and analyze data, and then trust it to make the right decisions, if they are to continue in their efforts to enhance the status of their profession and meet today’s increasingly complex challenges.
So how, if at all, will this great inversion of experts and algorithms come about? How can ECC managers get better results by being more truly data-driven? It’s going to take transparency, time, and consequences: transparency to make clear how flawed “expert” judgment can be, time to let this news diffuse and sink in, and consequences so that we care enough about bad decisions to go through the wrenching change needed to make better ones.
Why should emergency communications center (ECC) managers pursue evidence-based decision-making and problem solving for their centers?
The “standard” approach to problem solving in an ECC is similar to that taken by many other disciplines, particularly those in the allied public safety fields of law enforcement, fire protection, and EMS: fix blame rather than fix the problem. We do this because in many cases we have not accurately defined the problem. Many managers believe that this issue—not defining the problem—is the foundation for the “I think, I feel, or I believe” methodology for decision-making. It is a management approach that has been shown to be frustrating to management (because we keep re-managing the same or similar problems) and demoralizing to ECC staff (because the root causes are never identified and everyone, regardless of their individual job performance, is subjected to the “training fixes all problems” mentality of management).
Regardless of the scope and magnitude of a center’s operations, it is imperative that an ECC manager develop his or her knowledge, skills, and abilities—as well as those of their staff—to engage in evidence-based problem solving (EBP) and evidence-based decision-making (EBD). EBP and EBD can provide significant benefits to an ECC, including:
- Better methodologies for developing information for presentation to community policy makers and decision makers when requesting necessary resources, e.g., new technology, additional staffing, revisions to policy, etc.
- Better methodologies for problem solving, such that true root cause(s) are properly identified.
- Better methodologies for objective decision-making, such that each problem gets the best solution.
- Improved morale among staff because problem-solving is objective and seeks to identify all causative facts for a given problem. • Improved recruitment and hiring outcomes for new staff members because “your center’s reputation precedes you.” (People like to work where the incumbent staff is happy, well-trained, and properly cared for by management.)
- Improved customer satisfaction in the public safety agencies and communities for whom you provide services because: (1) changes and improvements to your internal processes are likely to result in improved service delivery; and (2) changes and improvements to your internal processes are better received by those customers when ECC management can provide the “why, what, when, and how” behind such improvements.
What has to happen in the future?
In order for data analysis to lead to better decision making, organizations must embrace some key data-based business approaches. A good place to start is to ensure that the managers in our ECCs have the requisite knowledge, skills, and abilities to engage in both evidence-based research and evidence-based problem solving. Call it statistical literacy if you like, but training and professional development for center managers should include the fundamental elements of data collection and statistical analysis.
Attaining this body of fundamental statistical knowledge will provide the ECC manager with the ability to meaningfully explore data to find new patterns and relationships (i.e., data mining)—for example, how the number of calls for service varies by time and day of the week. They can use statistical analysis and quantitative analysis to help explain why a certain result occurred.
One example might be looking at the impact that a particular type of severe weather event, e.g., an ice storm, has on the number of calls for service received.
The new computer and communications technologies that have been installed in many ECCs are vast repositories of potential data for data mining. The key word is potential: ECC managers and their staffs must increase their knowledge, skills, and ability (KSA) to use these systems. While there is much emphasis—and rightly so—on developing the KSAs of front-line personnel to use these technologies—e.g., CADS, GPS, Automatic Vehicle Locators (AVL), etc.—ECC managers need to take the education and training process to the next level. That next level is the ability to locate the right data, extract it, clean it, and organize it into useful management information.
Defining the next level
Many ECCs currently have performance level goals and objectives for their daily operations that include, for example, the number of 911 and non-emergency calls answered and processed; the number of calls for service dispatched to public safety agencies that they service; the processing time for 911 calls; and the processing time for a call for service to be dispatched. These are practical and important measures for an ECC, but they are only the “tip of the iceberg” when it comes to the potentially valuable data that’s now available for EBD and EBP.
Successful managers use Business Analytics (BA), or the methodological analysis of past performance to identify issues and outcomes for the future, to objectively evaluate and make improvements to their operations through automation and optimization of business processes. These data-driven managers view their data as an organizational asset and leverage it for continuous improvement. In order for that to happen, managers must ensure that they are collecting and using quality data, and that both they and their subordinates understand both the technologies and the organization’s commitment to data-driven decision making.
All data is not statistical
The applicable data that can be useful to enhancing the EBD and EBP capabilities of ECC managers should not be limited to statistical data that neatly fits into a spreadsheet. People are the most significant resource for any ECC, and there are many opportunities for ECC managers to collect and analyze data so that they can better match their resources (people and equipment) with their needs (answering and dispatching calls for service). Some specific areas in which data should be collected are:
Telephone lines and Operational Staffing. Information should be collected on the number of telephone lines and operational staff employed by the ECC: Telephone service providers should also be contacted to determine which data is currently available and which needs to be collected to allow analysis of operational staffing and telephone line requirements.
Personnel and Fiscal Management. Managers must establish standards for workstation assignment frequency and a method for tracking workstation assignments. They must document the process by which workstation assignments are made and the number of hours that individuals staff workstations, then ensure that the process is understood and followed so that good data is being collected on personnel requirements for the ECC. Finally, they should develop a process for documenting the reasons that workstation positions must be staffed with personnel working overtime shifts and document the process by which overtime is scheduled. It can also be beneficial to establish a method for tracking who is offered overtime and who actually works overtime shifts.
Operational Procedures. Data-focused managers establish a methodology for updating operational procedures and related documentation. This should include the ability for stakeholder organizations, e.g., public safety agencies served, to submit data.
Quality Assurance. It is also necessary to establish and implement quality assurance processes that measure customer and stakeholder satisfaction, ensure that operational procedures are followed, and identify areas for improvement. For many ECCs, this will be in addition to the quality assurance processes that they already have in place for their Emergency Medical Dispatch (EMD) program.
Training of Personnel. ECC managers have a fiduciary responsibility, as well as an operational responsibility, to ensure that their personnel receive cost-effective training that enables them to do their jobs safely, effectively, and efficiently. The data collected for ensuring such efforts should be part of a learning management system (LMS) that includes job descriptions, training and certification requirements for both entry-level and incumbent staff, and recertification requirements.
