The Leapfrog grading system may put too much weight on the wrong factors, according to a study published in the journal Medical Care by researchers at the University of Michigan in Ann Arbor. Thus, hospitals with less than stellar scores on even one or two issues might be better off not participating in the system, researchers say.
The researchers studied the Hospital Safety Grade system created by The Leapfrog Group, a nonprofit patient safety organization. The Leapfrog scores are used widely as a shorthand assessment of a hospital’s overall safety, but the University of Michigan team says that may overstate their value.
Most hospitals participating in Leapfrog’s Hospital Survey self-report are in compliance with most, if not all, of the “Safe Practices” addressed by the survey, but the researchers show that how a hospital performed on these measures had little in common with how that same hospital performed on independent measurements of hospital-acquired infections, or with whether the government had penalized it for high infection or readmission rates. (An abstract of the study is available online at: http://bit.ly/2nvPVvZ.)
“The Safe Practices part of the Hospital Safety Grade is based on whether a hospital self-reports that it has adopted certain protocols, and not whether those protocols are actually followed,” Jennifer Meddings, MD, MSc, the study’s senior author and an assistant professor of internal medicine at the U-M Medical School, explained in a statement announcing the study report.
She used the safe practices measure involving hand hygiene as an example. “It may be news to consumers that this measure involves hospital self-reports of having specific hand washing protocols in place, not how often healthcare workers are washing their hands.”
The researchers also found that hospitals can inadvertently hurt their grades, and give a worse impression of the hospital’s safety than is warranted, by how they report scores. A hospital’s Safe Practices scores count toward its overall Hospital Safety Grade based on a comparison with the average Safe Practices measure score reported across all participating hospitals, the report explains. That means hospitals receive little grade advantage by reporting high scores but are at risk of much lower grades if they report less-than-perfect scores.
Hospital Safety Grades combine data several national sources, including public data from the CMS and the CDC -- and for hospitals that choose to participate, the voluntary Leapfrog Hospital Survey.
Just over half of hospitals in the study completed the Leapfrog survey, which includes self-reporting of eight National Quality Forum Safe Practices measures. If a hospital reports on them, it accounts for nearly one-quarter of the hospital’s total grade. Hospitals that do not complete the voluntary Leapfrog Hospital Survey also receive a Leapfrog Hospital Safety Grade, calculated entirely using publicly available measures.
The researchers ran simulations to see how these non-participating hospitals’ grades might have changed if they had self-reported low, average, or perfect Safe Practices measures, while their infection rates remained the same.
They found that even if their actual performance on hospital infections is good, such hospitals likely would end up with lower Leapfrog Hospital Safety Grades if their performance on the self-reported Safe Practices was anything less than perfect.
Hospitals that self-report less-than-perfect Safe Practices scores are likely better off not completing the Leapfrog Hospital Survey, the authors concluded.
