Create tests measuring exactly what you want
Expert offers practical advice on writing tests
Competency testing is a difficult area for a quality manager to tackle because there are few standards and guidelines to light the way.
While the Oakbrook Terrace, IL-based Joint Commission on Accreditation of Healthcare Organizations (JCAHO) requires accredited agencies to provide competency testing, JCAHO leaves the details and definitions up to the provider.
"For the most part, you have to define it yourself and set your own standards, developing a program around it; that’s time consuming and difficult to do," says Marilyn Hanchett, RN, MA, CPHQ, director of performance improvement for Express Scripts Infusion Services in Maryland Heights, MO. Express Scripts is a pharmacy benefit management company, and Hanchett works in the home IV therapy division.
"At a point in time when most providers are very financially strapped, and resources are limited, you kind of start off at a disadvantage," Hanchett says.
Still, quality managers cannot shortchange the competency test-writing process, she adds.
"The biggest mistake I see people make is they say, OK, I have an employee who is an RN and I want to test her for competency,’ and then they jump from looking at the rule to wanting to write the examination," Hanchett explains.
This will result in the test writer skipping the important steps of writing a detailed analysis of the employee’s job and analyzing the employee’s expected skills level and knowledge of tasks in various categories.
Also, it’s not a good policy to use a generic competency exam for a licensed clinician. "What an RN is doing in St. Louis may be different from what an RN is doing in Chicago," Hanchett explains. "It ultimately needs to be tailored to that agency’s particular job description."
Plus, JCAHO expects agencies to customize their competency exams and correlate them with their job descriptions. So even if an agency chooses to buy a generic competency exam, it will need to be adjusted according to that agency’s particular needs and job descriptions.
So if a quality manager chooses to use a generic test without adjusting it, then the completed competency exam likely will not accurately describe a particular employee’s skills level, which is what the JCAHO surveyor will want to see, Hanchett says.
What makes a good question?
Quality managers and administrators who have little experience in test writing might contact a professional organization, such as the National Association for Healthcare Quality in Glenview, IL, to see when they will offer the next test-writing seminar or workshop, Hanchett suggests. "They teach you what goes into a good question, and it’s very hands-on information."
Hanchett offers these suggestions on how to write a competency exam that will fairly and accurately assess an employee’s skills and knowledge:
1. Write a thorough job description.
The job description is crucial because it will describe exactly what tasks are required of a particular job.
"A well-written job description can help you analyze the categories of duties, so when it comes time to develop a program and prepare an exam, the exam should tie in with the job description," Hanchett says.
For example, if a home care aide’s job is mostly to provide personal care or support for activities of daily living (ADLs), then 80% of the job description and exam should address those tasks, she adds.
In writing the job description, managers should consider whether the professional staff will function within a generalist model or a more specialized model. And keep in mind that if clinicians are expected to be generalists, then it’s much more difficult to write a competency exam, Hanchett says. "A nurse who is handling very complicated infusion therapy and chemotherapy at the patient’s home might never have touched any pediatric patients."
If the agency suddenly wants to send her to do a pediatric assessment, a quality manager has to wonder how thorough the nurse will be when dealing with a type of case she rarely handles.
Those types of possibilities should be addressed in the job description. If the nurse is expected to be competent primarily in infusion therapy and chemotherapy, then the job description should reflect that the person’s focus will be primarily in that area.
2. Write the exam.
The exam should be tailored from the job description, Hanchett says.
If the job description states that a particular job requires primarily Skill A, about half as much of Skill B, and a little of Skill C, then the quality manager could assign a percentage to each of those skills, i.e.: 60% of Skill A; 30% of Skill B; 10% of Skill C.
Next, take those percentages and write competency exam questions that fall into one of those three categories, with 60% of the questions pertaining to Skill A, and so on.
"That’s one way of organizing it to make sure you’re covering all the areas in the job description," Hanchett explains.
Now it’s time to decide what type of questions to write. The easiest questions to write and answer are the simple recall questions that test a person’s memory. Application questions are a little more difficult because they ask a person to figure something out, such as a mathematical equation, and the third type is analysis questions, which require the test-taker to user higher cognitive thinking skills in determining an answer. (See test-writing tips, inserted in this issue.)
"A good test should cover all three types of questions and cover all the content in the job description in proportion relative to the job role," Hanchett says.
The written exam typically is between 50 and 150 questions, with 100 being the average. The more complex the job description and the higher the level of skills for a particular job, the more questions are needed to assess competency.
Another simple rule to follow is that the competency exam should include mostly multiple choice questions. These are more time-consuming to write, but true/false and fill-in-the-blank questions are a less precise measure of competency, Hanchett says. (See competency item writing reminders, p. 41.)
Competency exams also should include an equally important second part, which is the performance-based part of the test. This is where a supervisor provides one-on-one, hands-on observation of how an employee handles various job tasks.
3. Assess the test results.
"You have to test the test and sit down and analyze how people performed on those questions," Hanchett says. "If you have people routinely doing poorly on the exam or falling short, then it’s entirely possible there is something wrong with that testing instrument."
To assess the test, a quality manager could subject it to a peer review process, giving it to other managers for comment. This is a little tough on the ego because there’s likely to be criticism of how various questions are written, but it’s one way to make sure the test is as objective as possible.
The ultimate assessment of a competency test is when it’s given to staff and scored. The scores shouldn’t be uniformly high, which may indicate the test is too easy. And there shouldn’t be a greater-than-expected number of poor results.
One check and balance of the written exam is the performance exam. Clinicians who do well on the written part of the test should also do well on the performance part, and vice versa. If a quality manager notices a trend that indicates the staff are doing well on the written part but not on the performance part, then this means there’s a problem with the written part of the test.
For example, suppose a nurse answered all the questions correctly about caring for an indwelling urinary catheter. But when tested in the field, the nurse couldn’t perform the task correctly. This type of discrepancy means there’s a problem with the test, Hanchett says.
"You may have to explore the content of those questions because maybe they’re too easy or they’re asking things that are not linked to the performance aspect of the job," she says. "The questions have to provide the springboard to actual performance and skills in patient care."
This is another reason why a good competency exam includes more than simple recall/memory items. Some of its questions should include trouble-shooting scenarios, such as, "You arrive at a home and saw this sort of problem with a catheter, so what would you do first?" Hanchett says.
"You’re not just testing the ability to remember, you’re testing the ability to think through and problem solve," she adds. "Granted, on a piece of paper you can only do that to a limited extent, but you’re building a platform of knowledge so that the next time the employees go into the field, they’re at least doing that from a stronger basis."
4. Retest staff.
After analyzing test results and revising questions that appeared to be too easy or unnecessarily difficult, a quality manager could retest employees to again assess the test’s objectivity and ability to assess competency. Or, at least, the quality manager could have the employees who failed to make the passing score retake the test after they have received additional training in their deficit areas.
"If a person doesn’t pass, this is a trigger for individual training, so you might assign a preceptor or do formal classroom training," Hanchett says. "The competency assessment is never intended to be a punitive exercise; it’s to benefit employees in terms of professional and paraprofessional development."
The passing score could be 90% or 80% to 85%, depending on the skill level of a particular job.
"In an ideal world, you want the scores to be as high as possible, and after that it is a strategic management decision," Hanchett says.
"Generally, the higher the level of technical skills, the higher the expectation level on the exam; so a generalist nurse might have a passing score at 70% to 75%, but an IV therapy nurse might have a passing score at 80% to 85%."
The second test could be exactly the same as the first one, or it could have the same components, mixed in a different order to avoid the problem of a person scoring well due to good recall of test answers. Quality managers also could change the phrasing slightly. n
• Marilyn Hanchett, RN, MA, CPHQ, Director of Performance Improvement, Express Scripts Infusion Services, 13900 Riverport Drive, Maryland Heights, MO 63043. Telephone: (314) 702-7881.
• Linda Krulish, MHS, PT, President, Home Therapy Services, 2205 Sugar Maple Court, Acworth, GA 30101. Telephone: (678) 521-5842. Fax: (770) 529-4913. E-mail: [email protected].
• National Association for Healthcare Quality, 4700 W. Lake Ave., Glenview, IL 60025. Telephone: (800) 966-9392. Fax: (847) 375-6320. Web site: www.nahq.org.
You have reached your article limit for the month. Subscribe now to access this article plus other member-only content.
- Award-winning Medical Content
- Latest Advances & Development in Medicine
- Unbiased Content