Choosing the right measurements is key
June 1, 2011 5 minutes read
Choosing the right measurements is key
How to measure what you measure
For some organizations, finding out where you fit in compared to other organizations is something they do regularly and well. But what they should really be doing is measuring how they compare to themselves in the past and figuring out a way to improve what they do now, says Steve Meisel, Pharm.D., director of patient safety at Fairview Health Services in St. Paul.
He gave a presentation on the problems with QI measurement tools framed like a "point-counterpoint" skit from old Saturday Night Live sketches at the November meeting for the Minnesota Alliance for Patient Safety to look at several ways of measuring safety and talked about their positive and negative traits.
"Everyone has flaws. There isn't a generically good, let alone perfect, tool that works well for everything," he says. "And depending on who you are, you might choose different metrics. Some might be demanded by an interest group, some might be self-assessments that ask you whether you do these 150 things or not." National Quality Forum, Leapfrog, CMS there are dozens of organizations that want you to measure things and show how you measure up against others. Meisel says none of it means your patients are safe. "It just means you do these things or do them well."
Best practices are a great idea. Doing them is good. But does that mean that you don't make mistakes? If you do all of them, and all of them well, Meisel says you can't promise that all your patients are safe, that you will have no bad outcomes or if you do none of them that you will have no good outcomes.
Self-assessments have a natural bias in them that leads to more positive results than are true. That said, they can be a great guide to figuring out where you need work. For instance, if a pharmacist wants to figure out the integrity of the medication system and does the Institute for Safe Medication Practices assessment tool, you might very well find after completing it that there are areas you need to work on, he says. And there are some great best-practices tools from the Association of periOperative Registered Nurses that could help you figure out if you are doing all you can to reduce wrong-site surgeries.
"One problem with measurement is an inability to differentiate error from adverse events and outcomes," says Meisel. "Most errors don't cause bad outcomes. You give two anti-acids not one. Nothing bad happens. But it is still an error. Some can cause harm, or cause waste and rework. That's different than safety. We haven't done a good job defining this."
People mix up adverse events and errors, Meisel says. "We have to report any serious event. But when it's released to the media every January, the headline is that so many errors hurt so many people last year. But some errors are wrong-site surgery. Some are physical assault of staff members. They aren't the same thing, but they are measured the same way."
We have to stop measuring to look for the absence of bad outcomes. "People will die in healthcare," he says. "There will be complications. What is safety is the knowledge that we are preventing all we know how to prevent, and are learning how to prevent those we don't currently know how to prevent."
Language is important in measurement. For some things, there are standard definitions what is an infection, for instance. However, another metric may not have just one definition. Pressure ulcers are fairly standard in how they are graded, but AHRQ has certain included indicators and CMS has others. In one, if you are a quadriplegic you are excluded; in the other, you are not. That can change the way your hospital looks when compared nationally with one data set or another. "In some respects, it doesn't matter what you choose as long as you use it for directional purposes. If you go up for one, you will probably be going up for all."
Everyone will choose something different, and everyone will gather data differently. "You are better off looking at internal problems what occurs most frequently and has the highest risk where you are. Develop your metrics, make sure they are outcomes-oriented, and check regularly internally," Meisel says. Indeed, he calls benchmarking a "prescription for mediocrity." You can say you are as good as everyone else, but if everyone else is mediocre, what good is that?
Another issue Meisel has with measurement tools as they currently are used is that a lot comes from administrative datasets. "The numbers are a reflection of coding more than of clinical care," he says.
Trigger tools are also problematic for him. The Institute for Healthcare Improvement has one that lists dozens of issues that you should look for in medical records reviews to see if an adverse event is associated with the trigger. An example might be if a patient was transferred to the ICU or returned to the OR. "The problem with that tool is that if you only do 20 records a month, if it happens less than 5% incidence, it won't get picked up. You tend to get the relatively frequent but less serious issues. You don't see wrong-site surgeries, but you probably will see nausea from narcotics that isn't life threatening."
Ask Meisel to dream up his ideal world of measurements and tools, and he says he would concentrate on the half-dozen things we know are big issues in safety like falls, wrong-site surgery, and retained foreign objects. "Develop a metric that will work for you that would also be a headline in a paper: We have done away with narcotic over-sedation or no wrong-site surgery in five years. Those are the metrics that are meaningful to the public."
Don't look at things that are internally focused on tactics and processes or are very obscure like doing a failure mode analysis. "Those are things we should do, but that's not what I want to report to the board," he says. "The same is true with other metrics that have roles but aren't important. Sure you can measure culture. But just because you have a positive culture doesn't mean you are safe. It means you have the capacity to embed process changes that others might not be able to do because culture isn't as positive. That's not a safety measure in itself, though. You still work on it, but not a great banner."
The toolbox of the quality professional has to have everything in it. You do the best practices, you have the good culture, you even do the self-assessments. But if you want to be patient-centered, you have to focus on the things that mean something to patient outcome and measure against the population that means something to your patients that is, yourself.
For more information, contact Steve Meisel, Pharm.D., Director of Patient Safety, Fairview Health Services, St. Paul, MN. Email: [email protected]
For some organizations, finding out where you fit in compared to other organizations is something they do regularly and well.You have reached your article limit for the month. Subscribe now to access this article plus other member-only content.
- Award-winning Medical Content
- Latest Advances & Development in Medicine
- Unbiased Content