Do report cards score with patients and doctors?
AHCPR report questions reliability of profiles
A new study sponsored by the U.S. Agency for Health Care Policy and Research (AHCPR) questions whether the current generation of "physician report cards" can really provide people with information on how well individual doctors manage specific health conditions and whether they can help people select doctors, or help doctors improve their own performance.
The study looked at how doctors in three types of medical practices — a large West Coast HMO, an urban medical school teaching clinic in the Midwest, and a group of private-practice physicians in New England — managed patients with Type 2 diabetes. This particular disease was selected for its prevalence in primary care practice and because the way doctors manage diabetes can affect outcomes in their patients.
The researchers found that the report cards, or physician profiles, for diabetes were unable to reliably detect true practice differences among doctors at the three medical practices studied. Differences in how the physicians managed their patients’ diabetes (e.g., their practice styles) contributed only 4%, at the most, to the overall variance in their patients’ hospitalization, office visit, laboratory use, and blood sugar level control rates. The difficulty in using these outcomes to evaluate physician performance was due, in large part, to the relatively small number of diabetic patients managed by each doctor.
An abstract of the study in the June 9 Journal of the American Medical Association states: "For profiles based on hospitalization rates, visit rates, laboratory utilization rates, and glycemic control, 4% or less of the overall variance was attributable to differences in physician practice and the reliability of the median physician’s case-mix-adjusted profile was never better than 0.40."
At this low level of physician effect, "a physician would need to have more than 100 patients with diabetes in a panel for profiles to have a reliability of 0.80 or better [while more than 90% of all primary care physicians at the health maintenance organization had fewer than 60 patients with diabetes]."
The abstract further observes, "For profiles of glycemic control, high outlier physicians could dramatically improve their physician profile simply by pruning from their panel the one to three patients with the highest hemoglobin A1c levels during the prior year. This advantage from gaming could not be prevented by even detailed case-mix adjustment."
"Our findings do not mean that we physicians should not be accountable for our patients’ outcomes, nor that attempts to evaluate the medical and quality-of-life outcomes of our care are futile because the average doctor doesn’t see enough patients for a specific condition to make analysis meaningful," says Sheldon Greenfield, MD, a co-author of the study. "Instead, the findings suggest that health plans and hospitals need to take a better look at the science they’re using to do report cards."
According to Greenfield, who is with the New England Medical Center in Boston, each of the physicians in the study would have had to have managed over 100 diabetic patients for the researchers to detect any meaningful differences among them. "Apart from hypertension, it is difficult to imagine that there would be enough cases per primary care physician to construct disease-specific profiles for almost any other chronic condition," says Greenfield, who adds that doctors sometimes receive evaluations based on as few as four patients.
"The science does not support such reckless use of numbers for judging physicians," according to Greenfield. "More accurate judgments could be made if physicians were evaluated in groups for the purpose of treating specific diseases."
The lead author of the study, Timothy P. Hofer, MD, of the VA Ann Arbor Healthcare System and the University of Michigan, says that another problem with the current method of profiling individual doctors is that the process can backfire if linked to powerful incentives not to have patients whose conditions are hard to manage and/or who incur high costs.
"The easiest way a doctor can avoid becoming a deselected provider,’ — that is, not have a bad report card — is to deselect,’ or drop problematic patients," says Hofer. "In our opinion, those implementing such profiling systems would be as much to blame as the doctors who deny care to the most vulnerable people needing medical help."
"Report cards on doctors may one day help Americans make informed choices," states John M. Eisenberg, MD, administrator of AHCPR. "But as this study so clearly shows, the current technology for profiling physicians is not reliable enough to detect practice differences with the number of patients a doctor normally sees for a given condition," he notes, adding, "We need more research to improve physician evaluation."
[For additional information, contact:
• AHCPR Public Affairs, Executive Office Center, Suite 600, 2101 East Jefferson St. Rockville, MD 20852. Telephone: (301) 594-1364. Web site: http://www.ahcpr.org.
• Timothy Hofer, MD, VA Center for Practice Management and Outcomes Research, P.O. Box 130170, Ann Arbor, MI 48113.]
You have reached your article limit for the month. Subscribe now to access this article plus other member-only content.
- Award-winning Medical Content
- Latest Advances & Development in Medicine
- Unbiased Content