Are physician profiles reliable measures?
Diabetic profiles need to include 100+ patients
A new study threatens to debunk a central notion of performance assessment: Namely, that measures can be used to determine which doctors are doing a good job of managing their patients’ care and which have the best outcomes.
In fact, individual primary care physicians don’t see a large enough number of patients with any single chronic disease to be rated on their outcomes, according to a study of physician report cards published in the Journal of the American Medical Association. And the physician impact on outcomes measures, such as blood glucose control for diabetics, is extremely small,1 says co-author Sheldon Greenfield, MD, director of the Primary Care Outcomes Research Unit of the New England Medical Center in Boston.
A study of 3,642 patients with Type 2 diabetes cared for by 232 different physicians at three diverse sites showed that only 4% or less of the variance in measures could be attributed to differences in physician practice. The researchers created profiles based on hospitalization and office visit rates, total laboratory resource utilization, and glycemic control.
To counteract this low level of physician impact, individual profiles would have to include more than 100 patients, the researchers found. At the West Coast staff-model HMO, where medical information systems allowed for detailed analysis, the average primary care physician had 29 diabetic patients. None of the physicians cared for more than 85 patients. The study also included physicians and patients at an urban university teaching clinic in the Midwest and a group of private practice physicians in New England.
At its heart, the complex statistical analysis outlined in the JAMA article addresses a matter of fairness. "There’s nothing wrong with collecting information on doctors and giving it to them [for quality improvement feedback]," says Greenfield. "What’s wrong is calculating a score and comparing it to other doctors [such as in a ranking or report card]."
Physician profiling also may result in a massive misallocation of resources, contends lead author Timothy Hofer, MD, MS, research investigator at the Veterans Affairs Center for Practice Management and Outcomes Research in Ann Arbor, MI, and assistant professor at the University of Michigan Medical School.
Comparisons of indicators may be more valid when they involve processes of care, such as immunizations or foot exams. But Hofer questions whether targeting physicians in an effort to reduce variation in care diverts attention from greater opportunities for improvement. "Most practitioners are now practicing in big corporate systems," says Hofer. "Those systems tend to introduce strong influences on what happens to patients."
Physician profiles often lead to targeted letters commenting on performance. Greenfield, an internist who only sees patients on Fridays due to his academic obligations, has been both unduly praised and criticized by health plans.
One plan sent a letter lauding him for spending less than his per-member-per-month allocation. Yet because Greenfield sees patients infrequently, he rarely sees those with serious illnesses. "My case mix is fantastic," he says. "I don’t have a lot of sick people who use a lot of resources. Is it really fair to penalize my colleagues who have sicker patients than me?"
In another case, he received a letter about four patients who needed mammograms. Only two had received them. If the health plan had calculated a rate based on that tiny sample, Greenfield would have been at 50%. One of the four women had refused a mammogram, and the other had temporarily relocated to Florida.
Waste of paper, waste of time
In fact, those letters based on ridiculously small patient panels prompted the statistical study of profiling, says Hofer. "A bunch of us were getting these sheets," says Hofer, an internist. "Being academic physicians, we have small practices. Your four patients used an average of $53 in drug costs. The average [among your peers] is $112.’"
Hofer, who is a member of the Blue Cross Blue Shield of Michigan Profiling Advisory Group, calls such mailings "a monumental waste of trees."
But Hofer and his colleagues had another concern as well. Physicians who wanted to dramatically improve their profiles could do so most easily by eliminating problem patients. "All you have to do is think about patients who don’t respond well to treatments or aren’t that compliant," says Hofer. His study found that the lowest-performing physicians could greatly improve their measure of glycemic control by eliminating one to three patients.
"If you’re designing an indicator, you want to make it easier for a physician to change their practice than to game the system," he says. Again, that means the indicator needs to reflect a high degree of physician impact rather than other influences, such as patient compliance with recommendations, he says.
At least in the short term, the JAMA article seems unlikely to affect programs designed to reward physicians who score well on certain indicators. At present, most published report cards focus on medical groups, rather than individuals.
For example, Health Net in Woodland Hills, CA, uses hearts in physician directories to designate medical groups that perform well on patient ratings. Those with fewer than 75 health plan members responding for that medical group were excluded, and comparisons were limited to the medical groups within each of five regions. (For more on report cards, see Patient Satisfaction & Outcomes Management, August 1998, p. 94.)
Other programs recognize physicians who achieve a stated standard. For example, the Provider Recognition Program of the American Diabetes Association recognizes physicians who perform well on at least eight of 11 clinical and patient satisfaction measures. Participating physicians must have at least 35 diabetic patients.
"They don’t use a comparison of one person to another. We just set a broad cutoff and recognize people above that," explains Greenfield, who is chairman of the Diabetes Quality Improvement Project, a measurement collaborative at the health plan level, as well as of the Provider Recognition Program.
The American Medical Accreditation Program, sponsored by the Chicago-based American Medical Association (AMA), will eventually include a performance assessment component. But its emphasis is on quality improvement rather than comparing and rating physicians, notes William Jessee, MD, former vice president for quality and managed care at the AMA and current president and CEO of the Medical Group Management Association in Englewood, CO. "You’d expect a quality physician to want to get some feedback from a reliable source on important process and outcomes measures that are relevant to his practice, regardless of payer source."
Patient vs. physician functions
But setting standards even on something as easy to measure as cesarean rates can be problematic, he says. "We simply don’t have enough reliable information and enough understanding of how much rates are affected by patient factors rather than physician factors to know where the cutoffs should be," he says.
Recognition programs can be designed to reward quality improvement without placing too much weight on comparative indicators, says Hofer. "When you have unreliable or imprecise measures, then you should set your incentives accordingly. You have weaker incentives the less precise your measures are," he says. "You can have stronger incentives the more precise your measures are."
Meanwhile, performance measurement programs will be on safer ground if they compare medical groups or health plans, say Hofer and Greenfield. The researchers also hope to see greater emphasis placed on establishing reliability and validity of quality measures.
"Quality of care needs to go forward," says Greenfield. "It just needs to go forward with good science."
Reference
1. Hofer TP, Hayward RA, Greenfield S, Wagner EH, et al. The unreliability of individual physician report cards’ for assessing the costs and quality of care of chronic disease. JAMA 1999; 281:2,098-2,105.
You have reached your article limit for the month. Subscribe now to access this article plus other member-only content.
- Award-winning Medical Content
- Latest Advances & Development in Medicine
- Unbiased Content