Harsh Criticism for New Report on ED Diagnostic Errors
March 1, 2023 6 minutes read
By Dorothy Brooks
New research that might have injected renewed vigor into improving diagnostic performance in the ED has instead prompted much uproar. In the emergency medicine community, that discussion has been overshadowed by biting criticism about the data and the methodologies investigators used to reach their conclusions — and what some are calling unfair blame placed at the foot of emergency providers.
The report in question was prepared by the Johns Hopkins University Evidence-based Practice Center for the Agency for Healthcare Research and Quality (AHRQ).1 The authors concluded that based on 130 million annual U.S. ED visits, the estimated diagnostic error rate in this setting is 5.7%. Further, the authors noted 2% of these encounters lead to misdiagnosis-related harms and 0.3% lead to serious misdiagnosis-related harms.
All told, the investigators estimated misdiagnoses in American EDs amount to as many as 7 million errors and 2.5 million harms each year. Also, perhaps not surprisingly, the investigators reported errors are most likely to occur when patients present with atypical symptoms. However, as soon as the report was released in mid-December, a letter penned by the American College of Emergency Physicians (ACEP) and nine other emergency medicine groups said they were deeply concerned about the research.
“After reviewing the executive summary and initial draft, we believe that the report makes misleading, incomplete, and erroneous conclusions from the literature reviewed and conveys a tone that inaccurately characterizes and unnecessarily disparages the practice of emergency medicine in the U.S.,” the groups wrote.2
Among other criticisms, the signatories found fault with how the report authors defined diagnostic errors, along with the applicability of the references cited to reach key conclusions. For example, the statement noted many report findings are based on studies that were conducted in European EDs where the training of emergency physicians often is substantially different than in the United States. In addition, the letter signatories noted malpractice data have been erroneously used in the report’s calculations to assess harm from missed diagnoses.
The comments concluded by stressing the repercussions from what the groups called a faulty report cannot be overstated. “It will irresponsibly and falsely alarm the public and potentially lead them to delay or even forgo treatment for time-sensitive emergencies, while also undermining the relationship between patient and emergency physician. The intended effect of improving patient care and increasing patient safety may, in fact, paradoxically result in greater harm,” the groups wrote.
Emergency medicine experts who have spent considerable time focused on research aimed at improving diagnostic performance in the ED expressed similar misgivings. For example, Hardeep Singh, MD, MPH, says he submitted comments about the report in March 2022 during the public comments phase.
“My comments outlined several methodological flaws, but the authors did not address them in the final report,” says Singh, chief of the health policy, quality, and informatics program at the Center for Innovations in Quality, Effectiveness, and Safety in Houston.
Singh notes he also shared his concerns about the AHRQ report with ACEP. Many of these concerns are outlined in an article in ACEP Now in which Singh talks about the authors making too many convenient extrapolations and cherry-picking data to make their conclusions.3
Joe Grubenhoff, MD, MSCS, the medical director of the diagnostic safety program at Children’s Hospital Colorado in Aurora, points to several specific flaws in the report that he says strain the credibility and the validity of the authors’ conclusions. For example, Grubenhoff notes the findings rely heavily on a few manuscripts written by the report’s lead author that were based on malpractice claims data. “By very rough count, the body, tables, and figures of the report reference [the lead author’s] work more than 60 times,” Grubenhoff says.
Grubenhoff also finds fault with the lack of involvement of emergency physicians in the study.
“To write a report that cites the ED, and by clear implication, emergency providers, as a source of serious threat to patient safety without involving more emergency medicine representation disingenuously reflects the way in which the medical profession generally self-monitors,” he says, alluding to the practice of specialties generally reviewing their own performance rather than that of other specialties. “The investigators in this report include only two emergency physicians ... only one of whom focuses on QI/patient safety and neither of whom have extensive publication records in diagnostic safety specifically.”
Grubenhoff acknowledges most of the technical experts involved with this work are emergency physicians, and emergency physicians served as peer reviewers. However, even some of these participants agreed with many of the criticisms cited by the multiorganizational letter.
For instance, Prashant Mahajan, MD, MPH, MBA, vice chair of the department of emergency medicine at C.S. Mott Children’s Hospital at the University of Michigan, says that as a technical expert for the AHRQ study, he expressed his reservations to the lead author before the report was published. In particular, Mahajan said he was concerned about the study design and the “inherent bias” of referencing so many of the lead author’s own papers.
Kenneth Michelson, MD, MPH, assistant professor of pediatrics and emergency medicine at Harvard, says he shares many of the same concerns expressed by other colleagues. “I’d also add that the report makes almost no mention of the issue of diagnostic test overuse or overdiagnosis (and no mention at all in the abstract/summary),” Michelson says. “There has been very little study of the risks of overuse and misuse of diagnostic tests, and there are real harms associated with test overuse, including direct harms (e.g., radiation from CT scans) and indirect harms (e.g., downstream tests and treatment results from the diagnosis.) This is particularly true in pediatrics.”
Further, according to Grubenhoff, the AHRQ report misses an important point about examining ED care using retrospective studies. “It is exceedingly easy to identify ED visits as the point where the error is made because they are discrete care events — say to a prolonged acute hospitalization or repeated PCP visits,” he says. “However, it does not follow that the location of and/or the providers of care are primarily or even uniquely responsible for diagnostic errors.”
Grubenhoff adds that any number of factors can make diagnosis particularly challenging by the time a patient presents to the emergency setting. For example, he notes the patient may not have had access to preventive or primary care.
Also, there may be poor medical record-sharing between hospitals, making it difficult for the emergency physician to see prior care was delivered. Sudden or time-sensitive events may afford the emergency provider little time to assess what happened before the patient’s arrival or to fully understand what the patient is experiencing. Additionally, Grubenhoff takes issue with the report’s estimate that one patient dies in each U.S. ED every week because of a diagnostic error attributable to ED care. “[That is] neither well-supported by the data contained in the report nor [is it] the clinical experience of most emergency physicians,” he argues. “Many deaths may occur in the ED, but it is a logical fallacy to conclude that the deaths resulted from care received in the ED.”
Grubenhoff agrees with the point made in the multiorganization letter, that emergency medicine leaders as well as health policymakers should be cautious in how they respond to the AHRQ report. “Emergency physicians are best positioned to understand the vulnerabilities in the diagnostic process unique to their practice environment that led to patient harm,” he says. “That the diagnostic process is particularly challenging in the ED is not news to emergency physicians or, frankly, the rest of the medical community. Generally, emergency physicians are quite conscientious of this risk.”
Further, Grubenhoff reiterates the point that finding the source of diagnostic error cannot be gleaned from malpractice claims data or administrative data alone. “Careful and structured chart review and analysis using validated instruments, [such as] the Revised Safer DX tool, are necessary to truly understand the burden of diagnostic errors in the ED.”4
ED Management tried to speak with the authors of the AHRQ report and obtain a response to the criticisms that have been leveled, but have not received any word as of press time. However, in a statement sent to ACEP Now, the authors acknowledged some of the criticisms they have received are fair but require a thorough response, while others are the result of misunderstandings that can be clarified. The authors also indicated they are preparing a response to the most frequently asked questions, which they intend to publish in the coming weeks. Furthermore, AHRQ indicated to ACEP Now that it plans to make available responses to all the comments that were submitted during the pre-publication phase of the report. While this generally takes three months, AHRQ said it hopes to expedite this process in light of the controversy and level of feedback that have accompanied the report.
REFERENCES
1. Newman-Toker DE, Peterson SM, Badihian S, et al. Diagnostic errors in the emergency department: A systematic review. Dec. 15, 2022.
2. Multi-organizational letter regarding AHRQ report on diagnostic errors in the emergency department. Dec. 14, 2022.
3. Grantham J, Dark C. Report on ED diagnostic errors sparks controversy. ACEP Now. Dec. 29, 2022.
4. Singh H, Khanna A, Spitzmueller C, Meyer AND. Recommendations for using the Revised Safer Dx instrument to help measure and improve diagnostic safety. Diagnosis (Berl) 2019;6:315-323.
New research that might have injected renewed vigor into improving diagnostic performance in the ED has instead prompted much uproar. In the emergency medicine community, that discussion has been overshadowed by biting criticism about the data and the methodologies investigators used to reach their conclusions — and what some are calling unfair blame placed at the foot of emergency providers.
You have reached your article limit for the month. Subscribe now to access this article plus other member-only content.
- Award-winning Medical Content
- Latest Advances & Development in Medicine
- Unbiased Content