Study questions credibility, value of clinician profiles
Recent evidence raises serious doubts about the reliability of individual physician report cards.1 Research investigator Timothy P. Hofer, MD, MS, at the Ann Arbor (MI) Veterans Administration Center for Practice Management and Outcomes Research led a study of profiles or report cards on diabetes care at three medical practices. He found that differences in individual practice styles contributed 4%, at most, to variations in rates of hospitalization, office visit, lab use, and blood sugar level control.
In a conversation with QI/TQM, Hofer explains, "We find that none of the measures are reliable at the physician level, but they might be reliable at the larger aggregate level."
Methodology of profile study
For research purposes, "reliability" is defined as the extent to which a measure produces the same result in repeated trials.
• Subjects: 3,642 patients with Type 2 diabetes cared for by 232 different physicians.
• Practice settings: Staff-model HMO, urban university teaching clinic, and urban private practice.
• Findings: After adjustments for severity of illness and other health problems, wrote Hofer and colleagues, "there’s not that much left to differentiate among individual physicians, and there was not that much difference among physicians."
To show any meaningful data on the effect of individual variations in practice, physicians would need to have more than 100 diabetic patients under their care. That is an unlikely scenario, Hofer says.
Given the average numbers in each physician’s panel, high outliers on profiles of glycemic control "could dramatically improve their physician profile simply by pruning from their panel the one to three patients with the highest hemoglobin A1c levels during the prior year. This advantage from gaming could not be prevented by even detailed case-mix adjustment," according to the report.
• Discussion of findings: In view of the small variations in individual practice, Hofer contends that "environment affects whether doctors do foot screening or eye screening."
Two problems arise with the rush to do individual physician profiles for public consumption, he notes:
1. It’s difficult to impossible to get reliable results given the present state of statistical knowledge.
2. The fixation on individual practice patterns drives people to "lose sight of the forest for the trees. We find that the forest view is more valuable at this time. In other words," says Hofer, "our study suggests that system adjustments are more important than looking for the bad apple physicians."
"Quality managers should be advocating systems approaches to physicians’ performance instead of singling out individual doctors," he adds. Although professional accountability is a "laudable goal," wrote Hofer and colleagues, and is increasingly used by health plans, profiling can add from $0.59 to $2.17 per member per month.
Researchers’ observations and warnings
• Physician profilers are urged to "consider that the application of profiles may foster an environment in which deselection of patients is the easiest way for physicians to avoid becoming deselected themselves [by third-party payers]."
• The variation attributable to patients’ prior utilization or experience is greater than the variation in individual physicians’ practices. Thus, it will generally be easier for physicians to change their profiles by avoiding care of more serious diabetic cases which have failed to respond to therapy or patients who do not follow treatment plans.
• While some specialists may have panel sizes large enough to allow for reliable profiling, "it is then important to ask if the differences between physicians are worth profiling."
Reference
1. Hofer TP, Hayward RA, Greenfield S, et al. The unreliability of individual physician report cards’ for assessing the costs and quality of care of a chronic disease. JAMA 1999; 281:2,098-2,105.
You have reached your article limit for the month. Subscribe now to access this article plus other member-only content.
- Award-winning Medical Content
- Latest Advances & Development in Medicine
- Unbiased Content