Educate the users of performance data
QI success hinges on reliable information
By Patrice Spath, ART
Brown-Spath Associates
Forest Grove, OR
The last phase of a health care performance measurement (PM) project is the step of analysis. Once guideline-based review criteria have been developed and the data collected from a representative sample of cases, the users of the information examine the measurement results. The committee involved in designing the original study is likely to include the primary users of the PM data.
During the analysis phase, the committee members will be looking for confirmation that the clinical practice guidelines were appropriately applied during each episode of health care services, and if not, where improvements might be warranted. In addition, there are secondary users of the same PM data. These secondary users include internal groups like the organization's governing board and external groups such as health plans and even patients themselves.
It is essential to educate the primary and secondary users of PMs if they are to accurately interpret the study findings. The amount of education and training required for data users is dependent on the extent of their professional and clinical backgrounds. For example, a group of urologists analyzing the results of a study of compliance with benign prostatic hypertrophy clinical practice guidelines would need much different education and training from a group of health plan administrators who may be reviewing the same data.
Those individuals given the responsibility of evaluating the quality and appropriateness of care should be knowledgeable enough about the care given to determine those factors. In addition, an adequate evaluation of performance rates requires the users to understand the criteria development process, study design, mechanisms for establishing standards, and potential uses of performance rates.
Before evaluating the study results, the responsible committee members must be knowledgeable in many different aspects of the PM process. These required educational elements include:
· The original intent of the study.
For example, what did the criteria development task group hope to measure and why?
· The origin of the medical review criteria.
This could include the clinical practice guideline source(s) used in developing the criteria, how reliably the criteria match the guidelines, and the strength of evidence for the guidelines.
· The data sources used for assessing compliance with clinical practice guidelines.
Among these sources would be record review, patient interview, survey, or computerized clinical or financial database, for example.
· The data collection methodology.
For example, the technique for selecting sample population, results of data validity, and reliability testing would be included in the methodology.
· Peer interpretation of the findings in addition to the raw data.
Nurses should provide the interpretation if the study focused on the performance of other nurses; neurosurgeons should provide the interpretation if the study focused on the performance of other neurosurgeons.
· Study limitations.
For example, unobtainable record documentation, poor survey responses, factors that may be inappropriately implicated in causation, lack of professional consensus for particular guideline recommendations, patient preferences that confound the results, and missing or unreliable computerized data elements would be typical study limitations.
This education should be provided to the data users by the same clinicians and quality management professionals involved in the criteria development, study design, and data collection phases. Training may be provided through oral presentations during the meetings when the data are evaluated or may be offered as written comments that accompany the PM report.
In addition to being knowledgeable about the processes of health care delivery under evaluation, the users of the data should understand statistical analysis techniques. This includes training in data display techniques and interpretation. The report of study results should give the users information about the actual numbers of cases meeting each criterion, numbers of cases not meeting each criterion, and numbers of cases meeting exceptions to each criterion (if exceptions exist). Clear, focused reports should allow the users to immediately determine the following:
· whether the expected activities are being carried out, and if not, why not;
· what significant issues were identified and resolved;
· what issues were not resolved;
· how the group reviewing the reports can help.
A variety of charts, diagrams, graphs, and tables are available for data presentation, and data users should be sufficiently trained to analyze study results presented in these formats. Prerequisite training for data users also should include fundamental descriptive statistical measures such as mean, range, frequency, and standard deviation. Industrial quality management techniques, such as control charts to evaluate process variation, also can be useful tools for data evaluators to understand.
Ultimate goal: Improvement of care
The users of PM data need more than just education in study design, limitations, and use of standards or other techniques to evaluate the results. They also need assistance in understanding how to use performance rates for quality improvement purposes. The goal of any PM process should be feedback to the clinicians being evaluated by the study and ultimate improvement of the activities of health care delivery. This is known as performance management.
Performance management is a continual process of establishing performance goals, providing feedback on existing performance, and providing reinforcement for performance. In addition, quality experts suggest that processes, not individuals, should be the objects of quality improvement. Therefore, performance management through measurement feedback should emphasize the analysis and improvement of health care processes - the "set of activities that go on within and between practitioners and patients."
The users of medical review criteria and PMs should be introduced to ways in which the criteria and/or the rates of conformance to criteria can be incorporated into the day-to-day process of health care delivery. By integrating the study findings with the process of care through such computerized techniques as patient care documentation systems, probability systems, and automated care recommendations, the medical review criteria developed for retrospective measurement purposes can serve as prospective or concurrent reminders of appropriate care. Noncomputerized methods for integrating guidelines into the health care delivery system include the use of clinical paths and case management, preprinted patient treatment forms or protocols, and process improvement techniques. Being knowledgeable in the ways in which medical review criteria and PM data can be used to improve the quality of patient care helps the users of the data act on the results.
Relative to the use of PM data, users also should be educated in the confidentiality of the information they are evaluating. To have others oversee their work is a threat to clinicians, especially if they cannot be assured that data will remain confidential. The ultimate success of any quality improvement effort hinges not only on correct, reliable, and accurate information about clinical quality, but also on the ability of the users to minimize fear of unwarranted disclosure. The users of performance data may wish to develop a policy on confidentiality of reports and information dissemination, much like some hospital-based quality management programs have adopted.
Legislation, profit motive drive PM efforts
Now, more than ever before, health care consumers are being offered a variety of data about the quality of health care services. Some of these PM efforts are being legislated. Several states now have mandated reporting requirements for providers to disseminate information on technical and functional quality to consumers. Other PM efforts are financially motivated, with many insurance groups and business coalitions regularly evaluating performance rates to identify which providers or clinicians appear to give high-quality patient care at a reasonable cost. Articles about health care costs and quality appear frequently in popular magazines and newspapers.
Whereas some secondary users of PM data have relatively sophisticated techniques for measuring health care quality and value, even the most advanced secondary users may not be fully informed about the complexities of health care services and how these variables affect the process and outcome of patient care. For example, the analysis of the hospital's mortality rate by the governing body requires that they understand the various patient characteristics that contribute to risk of death:
· the patient's physiologic status on admission to the hospital;
· the presence or absence of chronic illnesses and the burden of chronic illness on the patient's health status;
· the patient's functional status;
· the patient's preference about the goals of his or her medical care.
The quality of health care services has other dimensions besides patient outcomes, of course. Secondary users also analyze performance data that tell them about patient satisfaction with services, waiting time to see physicians, hours the facility is open, attitude of the staff members toward patients, and many other factors.
Educating secondary users of data
To correctly analyze PM data, especially the clinical factors, the secondary users of these data must be educated. And like the primary users, this education should be provided by those clinicians and institutions whose practices are being scrutinized. Educational components for the secondary users of health care PM data should include:
· the original intent/purpose of the study;
· the source of the medical review criteria and whether the criteria were developed from good clinical practice guidelines;
· the data collection methodology;
· peer interpretation of the findings in addition to the raw data;
· study limitations, provided by peer reviewers;
· examples of how the data can be used to assess and improve quality.
This education should be given in terminology that is understood by nonclinical people and that provides as much insight as possible into the complexities of health care services and patient care outcomes. Like the primary users of PM data, secondary users also should have a fundamental understanding of descriptive statistical measures such as mean, range, frequency, and standard deviation. Ideally, this education is provided to the secondary user in the body of the PM report to reinforce the user's understanding of the information being presented. More sophisticated secondary users, such as professional review organizations and payers, should explore various qualitative and quantitative data display and analysis techniques, such as statistical process control, and acquaint themselves with methods for making inferences about a population from a sample.
Without adequate education in the analysis of health care PM data, primary and secondary users are likely to misinterpret the information, or at the very least, receive little or no value from their analysis activities. Quality management professionals must lead the way in providing this education.
Related Reading
Donabedian A. The quality of care: How can it be assessed? JAMA 1988; 260:1,743-1,748.
Spoeri RK. "The Inspection of Data." In: Longo DR, Bohr D, eds. Quantitative Methods in Quality Management: A Guide for Practitioners. Chicago: American Hospital Publishing; 1991, pp. 79-123.
You have reached your article limit for the month. Subscribe now to access this article plus other member-only content.
- Award-winning Medical Content
- Latest Advances & Development in Medicine
- Unbiased Content