How to evaluate transfer of training to workplace
Observe what's actually happening
By Patrice Spath, RHIT
Brown-Spath & Associates
Forest Grove, OR
Quality and patient safety improvements often require changes in staff behavior on the job. A new procedure is put in place or an existing procedure is revised. It is relatively simple to evaluate whether staff have received training on the new way of doing things by measuring attendance at staff inservices. It is much harder to determine whether the training actually resulted in desired changes in practice. Evaluating transfer of training to the workplace is much harder to measure and often requires direct observation of what's actually going on.
The main advantage of direct observation is that work behavior can be studied in its natural setting, thereby providing a richer understanding of actual practices. Staff observations also can reveal conditions or problems that inhibit compliance with expected performance. On the negative side, direct observation is susceptible to observer bias. The very act of observation also can affect the behavior being studied. Even though observation has its drawbacks, it can be useful when:
- Performance monitoring data indicate results are not being accomplished as planned, and when implementation problems are suspected, but not understood.
- It is unclear why the process is poorly implemented or required documentation is absent.
- Details of an activity need to be assessed, such as whether tasks are being implementing according to expectations.
Tasks that are candidates for an observation review must be visible and of limited duration. In addition, the task must be predictable to allow for assessment of satisfactory completion. For instance, an observation study of caregiver response to clinical alarms could only be conducted if the responses are easily recognizable and the correct response for each type of alarm has been clearly defined. If these conditions are met, the observer can then view alarm responses and judge how often staff respond appropriately.
To ensure that data gathered during direct observations are valid, a systematic, structured process should be used. The quality of information derived from observational studies can be improved by following these steps.
Step 1. Determine the focus. Because direct observation reviews can be resource intensive, it is important to be selective. Look for only a few activities, events, or phenomena that are central to the evaluation. For example, suppose you want to study whether nurses are following proper procedures when infusing IV medications. Obviously you can assess a variety of activities — patient assessment, preparation of medication for infusion, pump programming, and monitoring. Narrow the focus of the study to one or two activities likely to generate the most useful information and insights.
Next, break down each activity, event, or phenomena into subcomponents. For example, if you decide to look at pump programming, prepare a list of tasks to observe such as entering drug dose and rate, manually calculating or selecting appropriate dose/rate from automated drug library, and verifying dose/rate data prior to pushing "run." You may also want observers to gather information that may be impacting staff adherence to best practices, such as interruptions, distractions, and environment factors (e.g., light, sound, room layout).
Step 2. Develop direct observation forms. The data collection form should list the items to be observed and provide spaces to record observations. These forms are similar to survey questionnaires, but observers record their own findings, not respondents' answers. Observation record forms help standardize the review process and ensure that all important items are covered. They also facilitate better aggregation of data gathered from various sites or by various observers.
Identify in advance the possible response categories for each study item, so that the observer can answer with a simple yes or no, or by checking the appropriate answer. To improve the quality of data, create closed response categories whenever possible to help minimize observer variation. It is also important to provide adequate space for recording additional observations.
Step 3. Select the study sites. Once the forms are ready, the next step is to decide where the observations will be carried out. A single site observation may be satisfactory if it is unique; for example, if only one unit or department is known to have deficiencies and the purpose of the evaluation is to discover why. However, study data obtained through single-site observations should not be extrapolated to other areas if the site is not representative of those other areas. As a rule, several sites are necessary to obtain a reasonable understanding of performance levels for tasks performed in more than one location.
Step 4. Choose the timing. Timing is important, especially when tasks or situations are to be observed as they occur. Wrong timing can distort the study findings. For example, during shift change or at times of lower staffing levels, caregivers may feel rushed and inadvertently skip a critical process step. If staff are observed primarily during these times, an inaccurate picture of performance may result. Conversely, don't avoid these periods for data collection purposes or the study findings may not provide an accurate determination of average performance. Try to strive for a representative observational sample from all times of the day and days of the week during which the task is performed.
Step 5. Pick observers. The best choice for observers are people who normally would be seen working in or near the study sites. Some hospitals have appointed quality or patient safety champions in every unit/department. These frontline staff occasionally are asked to periodically conduct observation reviews in their area. Supervisors or managers may be charged with gathering observational performance data. Support staff that are commonly seen in various units or departments also are potential observers. These staff include case managers, infection control practitioners, coding specialists, and patient educators. Although these support staff may not have expertise in the tasks they are being asked to evaluate, they can function as observers if the data-gathering form is straight-forward with mostly closed-ended questions. Specially trained observers from the quality department may be used; however, the presence of "outside" observers can generate some anxiety among staff being observed and also affect performance.
Regardless of who gathers the data for observation studies, it is important that those people being observed understand that the primary purpose of the study is to find out what kinds of problems may be preventing staff from adhering to best practices. The data should not be used to report on individual performance.
One of the advantages of direct observation is that staff can be evaluated in the environment where the tasks are actually done. People are just doing what they normally do and once in a while someone is observing their performance. However, you must always be aware of the possibility of the so-called "Hawthorne Effect" the fact that people usually perform better under observation because of the attention paid to them.
Quality and patient safety improvements often require changes in staff behavior on the job. A new procedure is put in place or an existing procedure is revised.You have reached your article limit for the month. Subscribe now to access this article plus other member-only content.
- Award-winning Medical Content
- Latest Advances & Development in Medicine
- Unbiased Content