Expert tips for improving metrics systems
Design systems for best measurement
When an IRB initiates electronic metrics collection and analyses as part of its quality improvement process, there will be obstacles to overcome. However, there are ways to make the transition smoother, including investing in designing the electronic system to obtain the measurements that will be the most useful, experts suggest.
“It often occurs that you decide to focus on a particular issue and you realize you didn’t design the system in a way that you can measure that issue,” says David G. Forster, JD, MA, CIP, chief compliance officer in the office of compliance at the Western Institutional Review Board in Olympia, WA.
“That’s a constant — developing new searches,” Forster says. “Programmers come up with new searches we didn’t think of in the original build.”
For instance, a routine data collection that IRBs should anticipate involves workload and its relation to staff and IRB meetings, says Daniel Nelson, MSc, CIP, director of the office of human research ethics and professor of social medicine and pediatrics at the University of North Carolina at Chapel Hill.
“How does the workload relate to the length of the IRB meetings?” Nelson says. “Some of these [measures] are done on a routine, ongoing basis with no real specific problem or issue we’re trying to address.”
Forster and Nelson offer these additional suggestions on how to best use IRB electronic metrics to make the transition to paperless data collection and analysis smoother:
• Anticipate an adjustment among IRB staff and investigators. “That’s predictable,” Nelson says. “We did inservice training and had campus-wide training systems.”
Even with training and plenty of time to prepare staff for the change, problems can occur. The UNC Chapel Hill IRB experienced the increased processing time, and this meant something was not working as well as expected.
“We needed to recalibrate some of our expectations,” Nelson says. “One of the reasons for having metrics and monitoring them on an ongoing basis is you have new employees and turnover, and it doesn’t take much to find that what was expected and normal for one group of employees may not be normal for a new group of employees; it’s not all about turnaround time.”
• Make certain your electronic system is compliant with regulations. “This is time consuming and expensive, but good to do,” Forster says. “If you are an IRB doing FDA-regulated research, it’s a good practice to have your system compliant with 21CFR, part 11 rules on electronic recordkeeping and signatures.”
To meet the requirements of part 11, IRBs have to validate your system, write test scripts for everything the system is supposed to do, and to test it to see if it actually does it, he explains.
“This makes your system more reliable and is a good practice for developing information technology systems,” Forster adds. “While the FDA is not proactively applying it to IRBs, it is in the manual that FDA uses to guide itself on FDA inspections, and it does include part 11 as one of the issues.”
• Test and validate the system. When an electronic system is designed or revised, there will need to be tests and validation.
“You have to come up with a list of user requirements and for every user requirement, every function your system does, you have to write a test,” Forster says.
“If you enter an investigator’s name and push this button, it should populate a field, and then you can run a test and say, ‘Yes, it did this,’” he adds. “It’s very time-consuming.”
But this level of validation is necessary to ensure accuracy in the electronic records, and accuracy is essential for compliance with federal regulations, Forster says.
It’s also important to train the electronic system’s users on how to enter data and to make sure the system is user friendly, Forster notes.
“We have quality checks we run,” he says. “For instance, we make sure we didn’t miss any continuing review anniversary dates, and we run that check once a week.”
When an IRB initiates electronic metrics collection and analyses as part of its quality improvement process, there will be obstacles to overcome.You have reached your article limit for the month. Subscribe now to access this article plus other member-only content.
- Award-winning Medical Content
- Latest Advances & Development in Medicine
- Unbiased Content