Do your satisfaction surveys tell you the truth?
Surveys and methods must be valid, reliable
Caution: Patient satisfaction data is only as good as the questionnaires and survey methods you use.
You won't see that warning on any of your patient satisfaction material. But you should consider it, nonetheless, if you use patient satisfaction scores to evaluate staffing levels, dole out bonuses, or revamp office processes, health outcomes experts say.
You need a valid, reliable survey and methods in order to collect accurate information. And soon, payers and purchasers may require you to use a standardized form that ensures the comparability of results, one national measurement expert predicts.
Statistically validating survey question and design is essential to your ability to accurately report results to payers. Taking this concept one step further, at least one national measurement expert predicts standardization of survey tools in the not-so-distant future.
Designing and administering a patient satisfaction survey may seem deceptively simple. You draft a few questions asking patients how they feel about their office waits, staff friendliness, physician thoroughness, and overall satisfaction. You hand them out at the checkout counter. Then you tally the results.
What can go wrong? "Questions can be so ambiguous that Jane Smith reacts one way, but Joe Brown reacts another way," says Darlene M. Trandel-Korenchuk, RN, PhD, grant and community outreach director for Nalle Clinic in Charlotte, NC. "You have people reacting to different meanings of a question if the question is not written in a clear manner."
She discovered that problem when she analyzed an in-house survey previously used by Nalle Clinic. The questionnaire asked patients to agree or disagree with the statement: "When I called for a non-emergency visit, I was able to get an appointment scheduled with the physician in a reasonable number of days."
But what is reasonable? A day? A month? Patients might even be confused about how to define a non-emergency visit.
"There are ways to weed out bad questions, through testing for reliability and validity," Trandel-Korenchuk says.
In other words, your survey should consistently measure the same concepts across different patients, and it should adequately and accurately measure concepts that define patient satisfaction, she says. (See explanation of statistical terms, p. 54.)
Beyond that, you need to ensure that your surveys and methods allow you to make valid comparisons across medical groups or providers.
Standardization of patient satisfaction tools that have been extensively tested will become increasingly important, says John E. Ware Jr., PhD, senior scientist with The Health Institute at the New England Medical Center in Boston and a nationally known researcher in health care measurement. Purchasers and accrediting bodies may require the use of standardized instruments or third-party contractors to conduct patient satisfaction surveys.
"We need to put our marks at the same place on the ruler," he says. "Otherwise we can't compare our measurement. It's as simple as that."
Most patients are satisfied. That means that you're measuring differences amid a tight clump of "very good" and "excellent" scores.
Therefore, the precision of your instrument is an important issue, Ware explains. Although very good sounds like a fine performance, patients are telling you something when they fail to rate you as excellent.
"Eight or nine out of 10 patients who rate you excellent are going to recommend you [to others]," he says. "That drops to 50% if they rate you very good. It drops to one out of 10 if they rate you good."
A short survey of general questions about satisfaction can give a pulse rate of your patients' experiences. But for specifics that will help you structure quality improvement programs and build a practice that engenders loyalty, you will need a more detailed questionnaire, advises Eugene Nelson, DSc, director of quality education measurement and research at The Hitchcock Clinic in Hanover, NH.
Do you want kudos or feedback?
In fact, patients who rate you well on overall scales may still reveal issues of concern when you ask about coordination of care, pain management, communication, or other focused topics, he says.
"The more general the question, the better you look," says Nelson. "The more specific the question, the more you find opportunities for improvement."
A survey should ask some actionable questions - those that address specific behaviors of physicians or staff, says David Radosevich, PhD, RN, director of the Center for Applied Research and Analysis at the Health Outcomes Institute, which is part of Stratis Health in Bloomington, MN.
For example, the Health Outcomes Institute survey asks if the doctor gave the patient an opportunity to ask questions and if the doctor gave instructions on how to use medication. "Those are things that provide a little bit more concrete feedback to the doctor," he says.
Sample size, response rate, and survey method are other issues that influence the quality of your patient satisfaction data. Will a receptionist be less likely to hand a survey to patients who seem unhappy at checkout - especially if patient satisfaction ratings are used to determine staff bonuses? To avoid that problem, you should provide surveys in a neutral manner, such as with paperwork the patient receives at check-in or through the mail, outcomes experts say.
Another rule of thumb: Don't lump together surveys that are sent to patients with those collected on-site. Different data collection methods, and even different response rates, can lead to differences in respondents. (For answers to some basic questions about measuring patient satisfaction, see story, p. 55.)
"This is not a research study, but it still needs to provide information that's valid," says Radosevich. "You have to use appropriate methodology and appropriately represent the data."
Why does validity matter?
These recommendations may sound unnecessarily complex. After all, most practices don't employ anyone with an expertise in statistical analysis. They're just trying to ask a more specific version of the question, "How satisfied were you with your care?"
Yet there are many ways to ask the question. Consider the experience of Trandel, who was a research consultant and on the faculty of the University of North Carolina in Chapel HIll when she offered to analyze the clinic's in-house survey and a vendor-supplied survey that the clinic was considering.
When she compared the two surveys to one developed by the Health Outcomes Institute, she found they were significantly less valid and reliable. The in-house and vendor-supplied surveys used positive wording - such as, "my wait in the office was reasonable today" - requiring patients to disagree if they were not entirely satisfied.
"For some people, it's very difficult to disagree, particularly with authority," Trandel-Korenchuk says.
An unduly rosy picture of patient satisfaction may be useful for marketing, but it won't pinpoint problems that may be the key to improving customer service and interpersonal relations. If positively worded questions are used in a survey, an equal number of negative items should be used to balance this affect.
In her study, she found that acceptability, or the relationship between the physician and patient, was much more important to overall satisfaction than accessibility. In other words, patients were willing to put up with office waits or other annoyances if they felt they had an excellent rapport with their doctors.
She also found little difference in the results based on different methods of surveying - such as phone vs. mail - and different response rates. Her conclusion: Medical groups are measuring the high end of satisfaction. "People choose providers they think they'll be happy with," Trandel-Korenchuk says. "If you don't like your provider, you're not going to stay with him or her." But that means greater challenges for practices as they try to create loyalty rather than just satisfaction.
"Maybe what we need to do is not just satisfy our patients but delight them, so they'll recommend our practice to others," she says.
The Hitchcock Clinic calls a random sample of inpatients and outpatients one or two weeks after hospital discharge or one to seven days after an office visit to ask them what delighted or disappointed them about their care. Phone interviewers ask patients if they had any "good or bad surprises." They also ask, "If you could tell us one thing we could do to improve care, what would it be?"
Such open-ended questions also could be included on a written survey, Nelson says. While the answers are impossible to quantify, they reach "into a different zone. These are things people didn't expect, but they met a need." Similar information could be gleaned from focus groups or observation, such as a mystery patient, Nelson says. (For more information on observation programs, see related story, p. 58.)
The underlying message of health outcomes experts: In your zeal to measure and rate patient satisfaction, don't neglect the methods that will lead to useful and valid information.
"We need to focus on the areas that matter to the patient," Radosevich says. "To me, [the physician-provider relationship] is the core of what we should be measuring when we think about satisfaction."
You have reached your article limit for the month. Subscribe now to access this article plus other member-only content.
- Award-winning Medical Content
- Latest Advances & Development in Medicine
- Unbiased Content