By Stacey Kusterbeck
Patients increasingly are using online symptom checkers and artificial intelligence (AI)-based platforms for guidance on whether to visit the ED. “Although these platforms have become widely available to patients and their family members, little is known about the accuracy, reliability, or safety of their use for patients,” says Tracy Madsen, MD, PhD, vice chair of research at Brown Emergency Medicine.
Madsen and colleagues evaluated two versions of the large language model ChatGPT (3.5 and 4.0), a WebMD symptom checker, and a symptom checker developed by Ada Health.1 “We looked at the accuracy and safety of some of the commonly used and recently developed symptom checkers to better understand how these platforms might be improved for future patient use, especially for patients trying to decide whether or not to seek care in emergency departments,” Madsen explains.
The researchers analyzed self-reported, previously collected data from 40 patients presenting to an ED, who had recorded their symptoms using the Ada symptom checker before seeing the emergency physician (EP). Research staff entered the data into the ChatGPT and WebMD tools. The diagnosis from the symptom checker tools was compared to the patient’s final diagnosis in the ED and compared with diagnoses and triage recommendations of three EPs, based on their review of the self-reported data from the Ada tool. The WebMD and Ada symptom checkers performed better than ChatGPT. ChatGPT 3.5 did have high diagnostic accuracy but had a high unsafe triage rate. ChatGPT 4.0 had the poorest diagnostic accuracy of the four systems but a lower unsafe triage rate and the highest agreement with the EPs.
Overall, the EPs performed better than the tools for accurate diagnosis of some common medical conditions that bring people to the ED. “Online symptom checkers may give patients unsafe recommendations as to where to seek care,” Madsen concludes. The proportion of cases where a tool suggested an unsafe triage decision ranged from 14% to 41%. In those cases, people might be advised by the tool to wait to seek care, or to go to settings other than the ED (such as primary care), when urgent evaluation in an ED is necessary. ED providers need to know that patients may be using ChatGPT and other web-based tools for guidance. “These tools, while potentially useful in some situations, need further study and validation in real patients before they are trusted as the primary source of guidance,” Madsen cautions.
Use of symptom checkers could lead to a patient reporting symptoms to the triage nurse that the patient may have not otherwise reported, but believed were important based on the symptom checker report, observes Kelly Gleason, RN, PhD, an assistant professor at Johns Hopkins School of Nursing. “Perhaps it could lead to some patients thinking they were misdiagnosed,” she says. “This highlights the importance of shared decision-making that includes the patient in the diagnostic process.”
REFERENCE
- Fraser H, Crossland D, Bacher I, et al. Comparison of diagnostic and triage accuracy of Ada Health and WebMD symptom checkers, ChatGPT, and physicians for patients in an emergency department: Clinical data analysis study. JMIR Mhealth Uhealth 2023;11:e49995.
Patients increasingly are using online symptom checkers and artificial intelligence-based platforms for guidance on whether to visit the ED.
You have reached your article limit for the month. Subscribe now to access this article plus other member-only content.
- Award-winning Medical Content
- Latest Advances & Development in Medicine
- Unbiased Content