Executive Summary
New research suggests that a popular 3M software program doesn’t clearly distinguish differences in care quality. The issue is important because the program is increasingly used to make payments to U.S. hospitals based on readmission rates.
- 3M says the study is flawed and the conclusion incorrect.
- The researchers concluded that either PPR flagged cases are not more preventable, or additional data collection is needed.
- The findings are based on a review of 100 randomly selected cases.
The Potentially Preventable Readmissions (PPRs) from 3M fails at distinguishing differences in care quality, including key factors involved in readmission, according to a recent report. One of the physician developers at 3M, however, says the study was improperly designed and the negative conclusion is not correct.
The report suggests that the software fails to distinguish between readmissions that are preventable and those that aren’t. The research was published online in BMJ Quality and Safety and an abstract is available online at http://tinyurl.com/o2cp5bq. Such a failure could be critical to hospitals because the PPRs are increasingly used to make payments to U.S. hospitals based on readmission rates, the authors wrote.
CMS posts data on 30-day readmissions for three common causes of hospital admissions: heart attack, heart failure, and pneumonia. Hospitals with high rates of readmissions are penalized financially and get less money from Medicare regardless of whether those readmissions could have been prevented.
In a bid to improve on the CMS measure and identify readmissions more likely to be preventable, 3M developed the PPRs measure, which is now increasingly used by U.S. state Medicaid programs for hospital payments. The 3M software identifies readmissions with diagnoses that are clinically related to those prompting the initial admission, to flag those patients whose readmission could have been avoided, and then generates hospital level rates of avoidable readmissions, taking account of population case mix and illness severity.
However, an unknown has been to what extent these pairings reflect quality of care problems and which readmissions are therefore truly preventable. Researchers led by Ann M. Borzecki, MD, assistant professor at the Boston University School of Public Health, Department of Health Policy and Management and the Boston University School of Medicine, looked at whether readmissions flagged as PPRs by 3M were associated with poorer quality of care than those that weren’t. They focused on Veterans Health Administration patients admitted to hospital with pneumonia, and readmitted within 30 days, between 2006 and 2010.
They reviewed the medical records of 100 randomly selected cases out of more than 11,000, to look at the quality of care these patients had been given while in hospital and after discharge, using processes of care derived from evidence-based data and a panel of clinical experts.
They were surprised to find that the quality of care among the 77 cases flagged as PPRs was slightly better than the 23 unflagged cases (total average scores of 71.2 vs. 65.8 out of 100), although this difference was not statistically significant. They found little information about the quality of care after discharge for flagged and unflagged cases.
Their findings lead the researchers to conclude that either PPR flagged cases are not more preventable, or that assessment of preventability requires other data collection methods to capture poorly documented processes.
“Among VA pneumonia readmissions, PPR categorisation did not produce the expected quality of care findings,” the researchers concluded. “Either PPR–yes cases are not more preventable, or preventability assessment requires other data collection methods to capture poorly documented processes (e.g., direct observation).”
3M submitted a response to BMJ Quality and Safety but has not learned whether the journal will publish it. Norbert Goldfield, MD, medical director for 3M Health Information Systems and one of the principal designers of the PPR classification methodology, provided a copy of the letter to Hospital Peer Review and elaborated on 3M’s concerns about the study. (For excerpts from the 3M letter, see the story later in this issue)
The letter begins by saying, “Unfortunately, the design of the study was based on a misinterpretation of the meaning of the PPR categorization as well as its intended use.”
The report was troubling to some quality leaders. In a linked editorial, physicians from Mount Sinai Hospital in Toronto, Canada, suggest that, “After years of intensive research to find an objective measure of preventable readmissions, it seems as imminent as the arrival of Godot.” Perhaps readmission rates are too crude a measure and aren’t patient-centered in the way some calculations assume, says one of the editorial authors, Christine Soong, MD, MSc, CCFP, assistant professor at the University of Toronto and director of the Hospital Medicine Program at Mount Sinai Hospital and University Health Network in Toronto. (The editorial is available online at http://tinyurl.com/plq4fmo.)
This latest research confirms that there is no “magic bullet” for identifying preventable readmissions, Soong tells Hospital Peer Review. She suggests that quality officers using the 3M software use caution in interpreting the data and not assume that the identification of preventable readmissions is correct or complete.
“The study suggests that the 3M tool is not sensitive enough to capture preventable readmissions, and if anything, it looked like some deemed non-preventable had better quality measures than those that may have been preventable,” she says. “Basically it’s telling us that we’re no further ahead in being able to identify readmissions that were preventable.”
As for 3M’s contention that the study was designed incorrectly, Soong notes that there is evidence suggesting that software systems using administrative data to calculate preventable readmissions fail to consider other important unmeasured factors that contribute to readmissions. Research suggests that hospitals with high “risk adjusted” rates of PPRs differed in many patient characteristics not included in the risk adjustments but which accounted for approximately 50% of the observed differences in probability of readmission.
“In other words, risk adjustment calculators such as those of 3M do not account for many patient factors that contribute to readmission, such as socioeconomic status, education, and functional status,” Soong says.
- Norbert Goldfield, MD, Medical Director, 3M Health Information Systems, Atlanta. Telephone: (770) 394-8800.
- Christine Soong, MD, MSc, CCFP, Assistant Professor at The University Of Toronto and Director Of The Hospital Medicine Program at Mount Sinai Hospital and University Health Network, Toronto, Canada. Telephone: (416) 586-4800, ext. 5464. Email: [email protected].
