Quality bonuses and physician profiles may be based on faulty survey data
Comparative clinical and survey data often lack validity
Medical groups and individual physicians are being held "accountable" based on flawed quality data or poorly administered patient satisfaction surveys, outcomes experts say. And those statistically invalid rankings may cost physicians money in incentive pay or lost contracts.
Patient satisfaction survey tools that lack appropriate validity and reliability are nonetheless being used to evaluate physician or medical group performance. Meanwhile, health plans are profiling physicians based on clinical measures that use inadequate sample sizes and don’t have sufficient reliability. (See related story, p. 87.)
One detailed statistical study of diabetes measures found that the physician impact on outcomes was dwarfed by other factors, such as treatment noncompliance, and that the influence of those other factors wasn’t eliminated by case-mix adjustment.1
"It’s astounding," says Sheldon Greenfield, MD, director of the Primary Care Outcomes Research Unit of the New England Medical Center in Boston, of the lack of reliability and validity testing of physician-profiling programs. "All of them should be subjected to the same rigor of any [evaluative] calculation."
Although patient surveying has a well-documented science, health plans and medical groups don’t always take care to use appropriate methods and tools. When the data are tied to financial compensation, the ramifications could be great.
"It may be frankly immoral to deny a person a promotion or a bonus on the basis of poor data," contends Raymond Carey, PhD, principal of R.G. Carey and Associates, health care measurement consultants based in Park Ridge, IL.
Health plans may base physician profiles on an unrepresentative sample of patients or may use a tool that wasn’t designed to gauge satisfaction with a specific physician, cautions Jerry Siebert, MA, president of Parkside Associates, a Park Ridge, IL-based consulting firm that specializes in patient satisfaction for health care organizations.
"You should insist on seeing a copy of the questionnaire and a description of how it was administered," he advises. "Many health plans will actually draw their group-specific or physician-specific satisfaction out of their annual member survey. You’ll find it doesn’t specify what a physician is being asked about or have any relationship to a [specific] visit.
"The plan has to make the assumption that the patient was talking about the same doctor that’s currently listed as the primary care physician [PCP] in their records, which may or may not be true," he says. "They may be answering those questions even though they haven’t seen their PCP in 18 months."
In fact, numerous pitfalls can lead to incomplete or inaccurate data.1 "A patient survey is a very valuable tool when it’s done well," says Carey. "It’s not worth anything if it’s not done well."
Is the sample truly representative?
Consider this true scenario of a health plan seeking to show a particular purchaser that it met target rates for patient satisfaction:2
Out of 1,000 subscribers from that company, the health plan contacted a random sample of 200 and asked them to complete a survey. However, rather than simply completing the survey on their own, the respondents were asked to attend a special session held at various times and dates during the workday.
Only 50 health plan members attended. The results appeared to be biased and not usable, and the purchaser ended up tossing them out and calling in a statistician. What happened?
When only 25% of the sample responded, that raised questions about the makeup of the nonrespondents. Survey samples need to be truly random and not biased, notes Ken Black, professor of decision sciences at the University of Houston, Clear Lake. Those who failed to show up may have differed significantly from those who did, he notes. "Any sizeable nonresponse makes the results open to question," he says. "Nonresponse is a problem in surveying, and an effort should be made to minimize its effects by encouraging sample response."
Another problem occurs when patient surveys from multiple physicians in a practice or group are combined without accounting for different sized patient loads. When patient surveys are used to profile or compare physicians, they should be based on proportionately representative samples, Black says. In other words, if John Smith has a panel of 1,000 patients and Bob Jones has 500 patients, then Smith’s sample size should be twice as large as that of Jones. An equal percentage of the doctors’ patients should be surveyed, he says. Random samples should be drawn from each doctors’ panel, not from the overall medical group or health plan population.
"Statistics are measures on samples. They’re not measures on the population," he says. "The whole game is to take some subset of the population and use it represent the whole population. The validity of everything that’s done in all these studies hinges on how representative the sample is of the population."
The larger the sample size, the greater the reliability of the results — or the less likely the differences among physicians are due to random variation.
There should also be a minimum number of responses per physician measured, which might mean surveying a larger percentage of patients of doctors with low volume, notes Siebert. Even if 50 patients respond from each physician’s panel, the confidence interval would be plus or minus 10%. A rating of 75% satisfied could actually be as low as 65% or as high as 85%.
That could be improved by using subscales, or several questions that relate to the same topic, such as satisfaction with access to care.
"If you had a child who came home with a history test he failed, and you looked at the test and it had one question on it, intuitively you would know this is not a good test," Siebert says. "The same thing is true of measuring customer perceptions. A group of questions is going to be better than a single question would be."
Until recently, at BroMenn Medical Group in Normal, IL, receptionists handed out surveys to patients as they registered for their visit. The survey, which was developed in-house, asked them to reflect on their most recent experience but didn’t relate to the current visit. Patients could fill it out before or after. The receptionists continued to hand them out until 20 surveys had been received for each physician.
But now that 20% of a physician’s bonus is based on patient satisfaction, the practice is moving toward using a vendor with an established, validated survey tool, says Paul Kribs, MBA, MHA, the medical group’s corporate leader.
Handing out questionnaires on-site can bias the results, says Siebert, who favors the mail-back method. "You know Mrs. Jones waited 40 minutes past the appointment time, she’s stomping her way out the door. Are you really going to stop her and say Will you please fill out this satisfaction survey before you go?’" he says.
"The most overtly dissatisfied patients are the least likely to be handed a satisfaction survey," he says. "Doing surveys on site is risky."
Patients also may feel uncomfortable and may worry about offending physicians with their responses to on-site surveys, says Carey. "If I really want your ideas, I give you something you can fill out and send back anonymously," he says.
Another common pitfall lies in the analysis of patient satisfaction results. Even with adequate sample sizes and response rates and valid survey tools, medical groups and health plans may make faulty comparisons, outcomes experts says.
Drawing conclusions based on two separate data points — such as results from two annual patient satisfaction surveys — can produce misleading results, says Carey. If your percentage of "very satisfied" patients goes from 40% to 43%, for example, that change may be insignificant.
"People don’t understand variation. They look at two numbers, last month’s averages and this month’s averages. Satisfaction goes up or down a few points. They don’t distinguish between random data variation."
More effective analysis looks at trends over time, perhaps from monthly or quarterly surveys, says Siebert. Such trends can be plotted on control charts that show the normal range of variation and reveal any outlying data points.
By understanding some of the pitfalls of survey measurement, physicians can demand that they be judged by high-quality data, says Siebert. For example, contract negotiations with health plans may include a request for information about the tools and methods of collecting comparative information on medical groups and health plans. "If I was on the receiving end of reports like that, which affected my reimbursement, I would want to very carefully analyze the methodology and the tool that was being used," he says. "I would not simply accept their word that it was being done properly."
You may also want to be armed with a response. "You should offer them countering data from a survey that you can prove was done properly and has documented reliability and validity," Siebert says.
References
1. Hofer TP, Hayward RA, Greenfield S, Wagner EH, et al. The unreliability of individual physician report cards’ for assessing the costs and quality of care of chronic disease. JAMA 1999; 281:2,098-2,105.
2. Carey RG. How to choose a patient survey system. Jt Comm J Qual Improv 1999; 25:20-25.
You have reached your article limit for the month. Subscribe now to access this article plus other member-only content.
- Award-winning Medical Content
- Latest Advances & Development in Medicine
- Unbiased Content