Ongoing Education, Outcomes-Focused Reviews Remain Key to Lasting Gains in Triage Accuracy
March 1, 2024 9 minutes read
By Dorothy Brooks
The triage score or acuity level assigned to patients at the beginning of their emergency department (ED) encounter typically sets the trajectory for how that visit will unfold in terms of when and where the patient will be seen by a provider and what resources will be required. Consequently, while triage assignments must be made expeditiously in the emergency setting, they significantly affect care.
However, a recent investigation into the accuracy of ED triage decisions when using the most common triage system — the Emergency Severity Index (ESI) — revealed that mistriage occurs in roughly one-third of patient encounters.1
Notably, this was an exhaustive study — researchers analyzed more than 5 million ED encounters across 21 hospitals. While the results suggest that overtriage occurred much more commonly (28.9%) than undertriage (3.3%), the researchers also found disparities in triage accuracy by patient characteristics such as race, age, sex, and socioeconomic status.
“We suspect these disparities are due in part to having a subjective triage algorithm, and that use of a more objective and standardized approach could help to decrease variation and mitigate biases,” explains Dana R. Sax, MD, MPH, the lead author of the research, an emergency physician with The Permanente Medical Group (TPMG) in Oakland, CA, and an adjunct investigator with TPMG’s Division of Research.
Sax tells ED Management that one of the big challenges with triage is making decisions with limited patient data. “We found that certain clinical characteristics, such as the use of high-risk medications, greater comorbid illness burden, and recent intensive care unit [ICU] stays, were associated with a higher risk of mistriage,” she says. “If triage clinicians were presented with this type of clinical data to assist with triage decision-making, we suspect that mistriage would decrease and that this could lead to more equitable triage decisions.”
There are myriad efforts underway to leverage machine learning, artificial intelligence (AI), and other types of technology to bolster the triage process with the kind of information that Sax suggests. For example, several hospitals within the Johns Hopkins Health System have replaced their ESI triage process with an approach that combines traditional nurse triage assessment with a clinical decision support (CDS) tool that makes triage recommendations.
Jeremiah S. Hinson, MD, PhD, an associate professor of emergency medicine at the Johns Hopkins University School of Medicine and co-director of the Center for Data Science in Emergency Medicine, explains that implementing the new tool begins with a thorough analysis of historical data regarding patient encounters at each hospital, specifically focusing on predictor variables for three outcomes:
- Patients who needed to be admitted to the hospital following their ED encounter;
- Patients who required ICU-level care or died following their ED encounter;
- Patients who required surgery or a procedure following their ED encounter.
From these data, developers can produce a predictive model that can analyze all the same variables, assess the probability a patient will experience any of the three outcomes, and then translate that information into a recommended triage score, Hinson says. (For more information, see the related story in this issue.)
How is this tool then used by triage nurses? “In real time, what happens is that nurses triage patients exactly as they have for 20 years,” Hinson explains. “We designed the tool on purpose so that the predictor variables that we take in are the exact same pieces of information that triage nurses [are accustomed to] gathering.”
For example, the nurse will gather demographic data, how the patient arrived in the ED, the chief complaints, and vital signs data. “Those are the primary predictors that go into the model. [They are what] the model is trained on,” Hinson notes.
However, in addition to the nurse inputs, the clinical support tool will surveil the electronic medical record (EMR) to see if a patient has active medical problems. Those will go into the model as well. “If there is no history at all, the model can see that, and so there will be a level of uncertainty it has about whether medical problems exist,” Hinson says.
At the moment that it takes the nurse to advance the computer screen where he or she has entered the patient data, the information collected by the tool will be processed using the machine learning algorithms that have been trained and optimized for each hospital. “Probabilities are generated and then translated into a recommended triage score [from 1 through 5], and then sent back to the EMR,” Hinson says.
The nurse will then see a recommended triage score along with an explanation as to why the specific level of acuity was selected for this patient. “For example, it tells what the major drivers of the prediction are, and then nurses can look at that and choose to agree with the tool … or they can put in a different number,” Hinson notes. “Most hospitals using the tool will allow workers to either agree with the tool or go one number above or one number below the recommended triage score, but not further away.”
Since the AI/machine learning triage tool was implemented at hospitals within the Johns Hopkins Health System in 2017, data show the approach has improved the identification of patients with critical illness, thereby quickening treatment for this group. Also, the new triage process has consistently decreased the proportion of patients allocated to triage level 3 while increasing the number of patients given lower acuity scores.2
While overtriage is not as big of a concern as undertriage, Sax and colleagues found that almost 30% of patients were overtriaged, and that the sensitivity of ESI to identify these lower-risk, lower acuity patients was only 50%. “Having a high rate of overtriage means that there are more lower acuity patients who are being assigned in the main ED stream, which ultimately slows the rooming and timeliness of care for sicker patients and likely leads to delays in care and lower quality for all patients,” he says. “Some studies have also suggested that overtriage biases downstream providers and may lead to overutilization of ED resources, including hospital admission.”
For all these reasons, Hinson believes algorithms like the one developed at Johns Hopkins might inform all clinical decision-making in the future — not just triage. In the meantime, he contends EDs could do more to improve triage accuracy.
“I understand that not everyone has the capacity to do this right now, but I think there is a component of this that everybody does have the capacity to do … and it’s about looking at outcomes,” Hinson observes. “We make a lot of decisions very quickly. We do it over and over and over, and yet we very rarely get information about what happened to our patients. Being able to feed information back to the user so that they can learn from prior mistakes or successes is critically important.”
Reviewing cases in this way may be more time-consuming in EDs that do not use a CDS tool like the one developed at Johns Hopkins. However, Hinson stresses it is important for nurses to know that a patient they triaged as a level 4 was rapidly transitioned to the ICU following their ED encounter, or that another patient who appeared to be fine and was discharged reappeared in the ED two days later with a serious diagnosis.
“Now that we have an electronic medical record where all of these data are captured and time-stamped, having [these cases] available to review is critical,” Hinson says.
While the triage process implemented within the Johns Hopkins Health System has essentially replaced ESI triage, other efforts are underway to use AI and machine learning to enhance the existing ESI triage algorithm. For example, the Emergency Nurses Association (ENA), which purchased the ESI program in 2019, has partnered with Burlingame, CA-based Mednition to see how these newer technologies can be used to arm triage nurses with additional information when making these decisions.
Similar to the tool used at Johns Hopkins, Mednition’s clinical support tool, dubbed KATE, was built using historical patient encounter data to help inform future decisions regarding ESI acuity. Studies have shown that ESI acuity designations made with this machine learning-driven clinician decision support tool are more accurate than the decisions made by nurses without the benefit of the CDS.3
Currently, more than 20 health systems are using the KATE tool to enhance their ESI triage decisions, but ENA leaders stress that EDs without this technology can improve triage accuracy.
The ENA has taken several steps to ensure that emergency nurses understand the ESI process and how to apply it at the point of triage. For example, the organization recently updated its ESI Handbook to reduce misinterpretations of the decision-making process, explains ENA President Chris Dellinger, MBA, BSN, RN, FAEN, director of Emergency, Trauma, and ICU Services at West Virginia University Medicine Camden Clark Medical Center. “We have examples of using the [ESI] criteria as well as what the high-risk presentations are,” she says. (More information is available at: https://enau.ena.org/Listing/e....)
For example, Dellinger notes that the new handbook includes more detailed explanations on how to identify ESI level 1 and ESI level 2 patients more accurately — the higher-level acuity designations. “We also have updated courses that include education on how to address our own biases … as well as special considerations, such as with respect to OB patients,” she says. “A lot of effort went into [those aspects].”
Research shows that providing regular, ongoing education on triage makes a difference in triage accuracy.4 Dellinger agrees that such education should be prioritized, but she also agrees with Hinson that a part of this education should include performance reviews. “I am the director of the ED where I work. We use the [ESI triage] training from ENA, but we also do ongoing case study presentations [regarding triage decisions] and do real-time feedback with our staff so that they understand,” Dellinger explains, noting that she pulls several charts every month to review with the triage nurses. “That is very important because that is how you learn.”
Sax and colleagues highlighted other potential targets for triage improvement. They found that certain clinical and visit characteristics, such as after-hours presentation to the ED, arrival via ambulance, recent ICU visits, and the use of certain drugs, were associated with the risk of mistriage. Such data might serve as a starting point for ED leaders who are seeking ways to improve triage accuracy. “Other EDs could apply our novel measures of mistriage to their ED data to see if they are associated with mistriage in their systems,” Sax says. “ED leaders could then use this knowledge to design QI [quality improvement] interventions addressing these challenges, including targeted education for triage clinicians or the development of protocols to flag these higher-risk patients at triage.”
Further, Sax explains that some EDs have implemented tactics to assist with the safe identification of lower acuity patients at triage. For example, Sax notes that many EDs use more objective lower acuity criteria — such as acceptable age ranges or specific presenting complaints — to assist triage clinicians, while others have placed physicians in triage to assist with decision-making.
Sax acknowledges that one important limitation of her study is that researchers still do not know the clinical and operational effects of mistriage. “We expect that mistriage matters a lot in some cases but may not be as important in many cases,” she says. “The next important step is to better understand when mistriage is impacting timely patient care and outcomes and overall ED resource utilization.”
Nonetheless, Sax suggests that EDs can endeavor to better understand triage quality and hone QI improvements in this area accordingly. She agrees that standardizing triage clinician education, specifically for high-risk situations, also might be helpful. “Further refining the ESI to make the branchpoints less subjective may also be beneficial,” Sax adds. “We suspect that in the future, more EDs can leverage the growing availability of comprehensive electronic medical records and advanced predictive analytics to assist and support clinicians. This transition will hopefully contribute to greater triage accuracy, equity, and downstream emergency quality of care.”
REFERENCES
- Sax DR, Warton EM, Mark DG, et al. Evaluation of the Emergency Severity Index in US emergency departments for the rate of mistriage. JAMA Netw Open 2023;6:e233404.
- Hinson JS, Levin S. Data-driven approach yields new approach for emergency department triage. ACEP Now. Dec. 6, 2023. https://www.acepnow.com/articl...
- Ivanov O, Wolf L, Brecher D, et al. Improving ED Emergency Severity Index acuity assignment using machine learning and clinical natural language processing. J Emerg Nurs 2021;47:265-278.e7.
- Butler K, Anderson N, Jull A. Evaluating the effects of triage education on triage accuracy within the emergency department: An integrative review. Int Emerg Nurs 2023;70:101322.
A recent investigation into the accuracy of ED triage decisions when using the most common triage system — the Emergency Severity Index — revealed that mistriage occurs in roughly one-third of patient encounters.
You have reached your article limit for the month. Subscribe now to access this article plus other member-only content.
- Award-winning Medical Content
- Latest Advances & Development in Medicine
- Unbiased Content