Part two of a two-part series
Top-ranked hospitals perform on par with peers
Rating methods weighted to operational excellence
(This month, QI/TQM looks at consumer-oriented health care quality information. What do quality surveys really measure? What kind of health care quality data do consumers use? Are physician profiles reliable? How much do consumer information efforts cost? It’s all here in the final installment of this in-depth report.)
Experts suggest that some of the highly publicized quality information for health care consumers reveals more about administrative efficiency and financial performance than clinical outcomes. One rating, the annual 100 Top Hospitals — Benchmarks for Success, published by Baltimore-based HCIA-Sachs, a health care information firm, was recently studied by a team of clinicians and statisticians led by Yale University (New Haven, CT) researcher Jersey Chen, MD, MPH.1
"Patients see billboards saying that a hospital was ranked in the 100 top hospitals, and they think they will get better clinical care," Chen explains, "but in reality, these hospitals are better performers financially." That data experts like Chen can make such a statement is more a reflection of the advances in measuring health care than attempts by the rating firm to mislead the public.
The top 100 rankings are derived from a hospital database of more than 800 data elements for more than 6,000 U.S. acute care and specialty hospitals. Among the performance measures used are risk-adjusted medical patient mortality and risk-adjusted postoperative infection index, severity-adjusted average length of stay and wage- and severity-adjusted average cost. (For details from the study, see "What do 100 top hospitals lists tell us, our patients?" p. 88.)
Statistical tools have advanced considerably since 1993, when the first 100 top hospitals survey came out. Statisticians can now, with some confidence, study the results and understand what they actually say. "Looking at the list of top cardiology hospitals, we started questioning whether the quality of clinical care was reflected in these studies," says Chen.
To answer the question, they matched the top 100 hospitals to similar facilities and compared clinical performance for the care of acute myocardial infarction (AMI).
On two clinical indicators, rates of beta-blocker use and mortality, the top 100 hospitals were comparable with their peers. However, the top 100 have lower lengths of stay and in-hospital costs. "That implies that the top 100 methodology is statistically good stuff, but most of the criteria are operational and financial. It better captures a hospital’s financial performance rather than clinical outcomes," he points out.
Chen and his colleagues conducted a similar investigation2 of "America’s Best Hospitals," a list published annually by U.S. News & World Report. This list is derived from equal weightings on: (1) mortality rates, (2) technological capabilities and nursing care, and (3) reputation. For reputation ratings, U.S. News surveys a random sample of physicians. Chen’s study finds lower 30-day mortality rates among elderly AMI patients admitted to America’s best hospitals. The research team concludes that a substantial portion of the survival advantage may be associated with the hospitals’ higher use of aspirin and beta-blockers.
The top 100 list could mislead the unschooled observer, Chen notes. "Although we are not implying that rating organizations deliberately set out to mislead the public or the health care industry, we have to ask who would value this information," he contends. Patients want the best clinical care. An HMO or managed care organization, on the other hand, might be equally concerned about an institution’s fiscal performance. "That’s not a disjoint," Chen notes. "It’s the dichotomy between who’s paying and who’s receiving the care."
Some quality specialists would raise the point that excellence in health care should be judged by clinical, customer satisfaction, and fiscal criteria. (For a related article on this point, see QI/TQM, June 2000, p. 66.)
With discovery of the operational and fiscal emphasis in the survey methodology comes the challenge for a top 100 hospital to explain to its constituents what the acclaim represents. "There’s no easy answer," concedes Chen. "The top 100 hospitals would naturally promote themselves as award winners. If someone crowns them as the best, who are they to argue?"
Besides, he reminds us, scientists have just recently had data sets to determine what the rankings really measure. However, one top 100 hospital is already struggling with that challenge.
The view from a top 100 system
The Beaumont Hospitals, including Beaumont, Royal Oak, MI, and Beaumont, Troy, MI, have made the 100 top hospitals listings for the past three years and two years respectively. In short, "We use caution in publicizing the recognition," says director of marketing and public affairs Mike Killian.
"We’re gratified to have the ranking," he explains, "but we have to be careful not to indicate that we’re clinically better in these areas than we might be. While the statistical analysis [used in the rating] is good, the criteria are administrative rather than clinical measures. I’ve been cautioned by our clinical leaders not to portray the ranking as a pure quality measure."
According to Killian, Beaumont Hospitals uses the citation to show managed care organizations how it runs its business. Top 100 status also attracts the news media — a mixed blessing, he notes. "I have a problem with their statistical illiteracy," he admits. He continually attempts to balance the favorable exposure with an honest interpretation of how consumers should read the findings.
He stresses that over time, comparisons of expected to actual results more truly reflect an organization’s performance than one year’s data. With certain surgeries, for example, a hospital could have an extraordinary case that skews figures. But aggregate data over several years would capture the typical performance.
Chen notes that many hospitals use honors such as top 100 standings to reinforce their employees as well as improve their community images.
As Chen’s study demonstrates, the rankings are like spotlights that shine on specific points of excellence in organizations on the list. That’s not to discount the nonranked, however. "The top 100 hospitals could be improved, and the peers are not necessarily inferior. It’s not about gathering awards," he says.
Killian agrees. The business of measuring health care quality is complex and as yet nonstandardized. In such an environment, he warns, "Caution is more important than accolade."
Some of the experts who spoke with QI/TQM contend that providers and payers are the real end users of rankings, profiles and report cards. Consumers don’t use them a great deal, according to Timothy P. Hofer, MD, MS, research investigator at the Ann Arbor (MI) Veterans Administration Center for Practice Management and Outcomes Research.
Consumers are interested in different indicators than the measures reflected in formal data presentations. A possible exception would be accessibility report cards that indicate time to first available appointment, and wait times at the provider’s office. (For more on consumer interest in and use of health care quality data, see "Are health care data used for consumer choices?" p. 91, and "Usefulness: The bottom line for consumer data," p. 92.)
As for the other value of rankings and report cards, Hofer says, "Perhaps it keeps us providers honest. Even if consumers don’t use them a great deal, just knowing we are being measured is an incentive to keep doing better."
Sarah Loughran’s experience confirms that point. As senior vice president of content for Healthgrades.com in Lakewood, CO, she often hears from facilities concerned about their standings on this consumer-oriented health care rating and information Web site.
"They’ll ask how we compile our ratings and how they can make their ratings better," she says. Raw data for the company’s ratings come from MEDPar (Medicare Provider Analysis and Review for clinical indicators) statistics and state all-payer files made public by 18 states.
Loughran describes a twofold effect from ratings such as Healthgrades.com: They supply information to consumers and nudge health care organizations to improve performance. There’s a strong parallel to the auto industry in the late 1970s and early 1980s, she says. "When J.D. Power put out information about the inferiority of U.S. cars compared to Japanese autos, the U.S. auto industry had a fit. They accused J.D. Power of being bought off by the Japanese, but ultimately, it resulted in phenomenal improvements in U.S. autos."
J.D. Power and Associates in Agoura Hills, CA, (www.jdpower.com) studies quality, performance and customer satisfaction for industries, including auto manufacturers.
While less than perfect, Loughran adds, measuring efforts such as Healthgrades.com and top 100 ratings spur hospitals on to find more accurate methods of monitoring our performance.
Reference
1. Chen J, Radford MJ, Wang Y, et al. Performance of the 100 Top Hospitals’: What does the report card report? Health Aff 1999; 18:53-68.
Need More Information?
For more on interpreting and designing consumer information services and tools, contact:
- Jersey Chen, MD, MPH, Researcher, Yale University, 333 Cedar St., Room IE61-SHM, P.O. Box 208025, New Haven, CT 06520.
- Timothy P. Hofer, MD, MS, Veterans Admini-stration Center for Practice Management & Outcomes, Veterans Affairs Ann Arbor, P.O. Box 130170, Ann Arbor, MI 48113. E-mail: [email protected].
- Mike Killian, Director of Marketing & Public Affairs, Corporate Administration, Wm. Beaumont Hospital, 3601 W. 13 Mile Road, Royal Oak, MI 48073. E-mail: [email protected].
- Sarah Loughran, Senior Vice President of Content, Healthgrades.com, Lakewood, CO. E-mail only: [email protected].
- Patrice Spath, Brown-Spath & Associates, 2314 19th Ave., P.O. Box 721, Forest Grove, OR 97116. Telephone: (503) 357-9185. E-mail: [email protected].
Check the following Web sites for consumer information on health care providers as well as suggestions on designing information services:
- www.brown-spath.com, Forest Grove, OR. Free original health care quality and resource management articles, and other services.
- www.healthgrades.com, Lakewood, CO. Free ratings of hospitals, health plans, and nursing homes nationwide; users may create customized lists based on quality criteria.
- www.100TopHospitals.com, Baltimore. Lists of the top 100 hospitals as well as other data products and services from HCIA-Sachs.
- www.usnews.com, Washington, DC. Lists annual rankings of America’s best hospitals and related articles.
- www.cyberdialogue.com, New York City. Posts survey results on consumer use of Internet resources, including health care information sites.
You have reached your article limit for the month. Subscribe now to access this article plus other member-only content.
- Award-winning Medical Content
- Latest Advances & Development in Medicine
- Unbiased Content