What are patients thinking when answering surveys?
'Cognitive testing' helped shape new CAHPS
What is the best way to ask a question and what information will it give you? Developers of the CAHPS survey (Consumer Assessment of Health Plans) believe they have found the answer through extensive cognitive testing - or interviews of survey participants.
In cognitive testing, researchers first administer the questionnaire, then quiz participants on how they interpreted the questions and why they gave the answers they did.
One major finding influenced CAHPS: Patients feel more comfortable describing what happened during an office visit than evaluating the aspects of that visit.
"It's a very humbling experience to take an item to a consumer and listen to them think out loud as they answer the question," says Jim Lubalin, PhD, director of the Washington, DC, office of the Research Triangle Institute (RTI) and a principal investigator in the CAHPS project.
"You have to think about the cognitive complexity of the task you're giving people," he says. "Often, it's way beyond their confidence to do it very accurately. Reporting is something people are able to do with some consistency and accuracy."
The CAHPS team also considered the consumer perspective when they favored reports as "a better metric for comparing the health plans," says Charles Darby, MA, CAHPS project officer for the Agency for Health Care Policy and Research in Rockville, MD. "If this person says she was satisfied, she may have a lower tolerance or a higher tolerance for this problem than I do."
Yet even with cognitive testing, choosing the appropriate wording for questions is hardly simple. Consumers may prefer to answer more objectively about what happened to them - but can they actually remember their experiences?
"Can people report what happens to them reliably?" asks Christina Bethell, PhD, MBA, MPH, director of accountability measurement for the Foundation for Accountability in Portland, OR. "I think there's some question about that. With ratings, you figure people can always respond about how satisfied they are."
On the other hand, health care ratings often are plagued with a sort of grade inflation. Patients tend to give their doctors high scores. "People say they are satisfied, but when you talk to them in a focus group they have all these issues and problems that come up," Bethell says.
John E. Ware Jr., PhD, executive director of the Health Assessment Lab at the New England Medical Center in Boston and a leading researcher in outcomes measurement, contends that the CAHPS team relied too heavily on cognitive testing. In the final analysis, he says, questions should be chosen based on their "empirical value" - how well they measure what they are designed to assess.
"I can predict what plans are best meeting the needs and expectations of the public five to 10 times more accurately with the same number of specific ratings than specific reports," he says. "People may be a little bit more uncomfortable providing 10 specific ratings than 10 specific reports. But the missing data rate is the same [on surveys] so they do it."
CAHPS developers considered how easily consumers could read and understand the wording, as well. For example, a literacy expert reviewed the questions for words that would stump some less able readers.
The excellent-to-poor scale also may be interpreted differently by Spanish-speakers than by English-speakers, according to the cognitive testing, says Ron D. Hays, PhD, professor of medicine at the University of California in Los Angeles and senior scientist at RAND Health Program. Instead, CAHPS uses a 0-10 scale on its rating items.
The survey doesn't ask at all about patients' perceptions of their physicians' technical quality. "Consumers are not necessarily the best source of information about how technically competent doctors are," says Hays.
CAHPS is still collecting and analyzing consumer input about the survey - but now the focus is on how to report the information obtained from the questions. "The early findings show they can't handle all the information we try to give them so we're streamlining the reports to make it more understandable," Lubalin says.
You have reached your article limit for the month. Subscribe now to access this article plus other member-only content.
- Award-winning Medical Content
- Latest Advances & Development in Medicine
- Unbiased Content