By Greg Freeman
Recent research suggests large language models (LLMs), a form of artificial intelligence (AI), may be useful in improving the quality of emergency medicine (EM) handoff notes, but it also illustrated shortcomings of the current technology.1
At some point, the technology may help reduce the documentation burden on emergency physicians.
The study notes that “EM-to-inpatient (IP) handoffs are associated with unique challenges, including medical complexity, time constraints, and diagnostic uncertainty; however, they remain poorly standardized and inconsistently implemented. Electronic health record (EHR)-based tools have attempted to overcome these limitations. However, they remain underexplored in emergency settings.”
Despite limitations, the results of the study are encouraging, says lead author Vince Hartman, CEO and co-founder of Abstractive Health in New York City.
“It’s something to be very excited about. Emergency medicine physicians, almost generally, all across the country, have a process where they communicate with internal medicine via telephone and or in person. Oftentimes, emergency medicine physicians don’t write [emergency medicine] handoff notes,” he says. “It was a unique workflow at this hospital where they had structured and created these handoff notes because of the benefit to the patient and reduction in safety risk for the patient. But oftentimes, across the country, the physicians just don’t have the time.”
The study was conducted at an urban academic 840-bed hospital and involved 1,600 EM patient encounters that led to acute hospital admissions. Researchers used a handoff note template similar to the hospital’s manual structure with two LLMs to produce EM handoff notes.1 The type of LLMs used in the study were not necessarily the most robust and advanced models available because of the hospital’s data security restrictions, Hartman says.
A clinical review of 50 handoff notes assessed completeness, readability, and safety to ensure their rigorous validation. The results indicated that LLM-generated summaries demonstrated greater lexical similarities, higher fidelity to source notes, and provided more detailed content than their human-authored counterparts.1
But on one scale, the LLM-generated summaries scored lower in terms of usefulness, completeness, curation, readability, correctness, and patient safety. Clinicians identified potential safety risks that included incompleteness and faulty logic.1
“Despite these differences, automated summaries were generally considered to be acceptable for clinical use, with none of the identified issues determined to be life-threatening to patient safety,” the researchers concluded. They noted that, “although most LLM-generated notes achieved promising quality scores between four and five, they were generally inferior to physician-written notes. Identified errors, including incompleteness and faulty logic, occasionally posed moderate safety risks, with under 10% potentially causing significant issues as compared to physician notes.”
Hartman says the technology has the potential for encouraging greater use of EM handoff notes.
“It’s like providing a tool that most physicians to date have viewed that would be too much time and burden for them to do, but, using LLM, they’ll be able to improve the patient experience workflow by creating an automated handoff structure,” he says.
Source
- Vince Hartman, CEO and Co-founder, Abstractive Health, New York City. Email: [email protected].
Reference
- Hartman V, Zhang X, Poddar R, et al. Developing and evaluating large language model–generated emergency medicine handoff notes. JAMA Network Open. 2024;7(12):e2448723.
Recent research suggests large language models, a form of artificial intelligence, may be useful in improving the quality of emergency medicine handoff notes, but it also illustrated shortcomings of the current technology.
You have reached your article limit for the month. Subscribe now to access this article plus other member-only content.
- Award-winning Medical Content
- Latest Advances & Development in Medicine
- Unbiased Content