Protecting research participants is the IRB’s core mission, but members often fail to measure how well they do it. To learn more, researchers interviewed 43 IRB stakeholders (i.e., leaders in research ethics oversight, study investigators, patient advocates, policymakers, and research sponsors) on how they viewed key features of IRB quality and how to measure those features.1 The researchers also interviewed 20 IRB directors on how they currently define and measure IRB quality, and how satisfied they were with that approach.
In the stakeholder group, participants struggled with how to define IRB quality, and how to identify ways to measure it. The IRB directors offered definitions of “quality” based on what their institutions currently measure, and were largely satisfied with these approaches (e.g., efficacy tracking, hearing feedback from researchers and board members, and audits of internal processes). When asked to identify core elements of IRB quality, both groups talked about efficacity, qualifications of IRB staff, compliance, and facilitating research.
Notably, IRB directors did not name participant protection or thoughtful review. Also, interviewees often struggled to define and measure quality and effectiveness. Researchers suggested IRBs adopt metrics specifically on participation protection outcomes to help IRBs to focus on their core mission. “We have to always remember our purpose: participant protection and high-quality deliberation. These are the areas where we have to focus our attention, resources, and measurement,” says Holly Fernandez Lynch, JD, MBe, lead author and assistant professor of medical ethics at the University of Pennsylvania.
“Measuring how effective IRBs are in protecting participants is a hard task,” adds Jon Newlin, CIP, director of operations for the Human Research Protection Program at The Feinstein Institutes for Medical Research in Manhasset, NY.
Accreditation from the Association for the Accreditation of Human Research Protection Programs, a history of uneventful routine audits by federal regulators, lack of FDA Form 483s documenting conditions that inspectors believe may violate regulations, and lack of FDA warning letters can be signals of high-quality IRBs. These boards also can examine how well they comply with federal regulations. “That is how most IRBs evaluate themselves, and how external entities evaluate IRBs. But it’s more challenging to empirically show that following those regulations protects participants,” Newlin says.
The Government Accountability Office used a unique strategy to measure IRB effectiveness. “They submitted a bogus study, purposefully written to be unethical, to three independent IRBs and waited to see what happened,” Newlin says.2
One of the three IRBs approved the study. “Such IRB ‘pressure testing’ is cumbersome to perform,” Newlin admits.
It is easier to perform that approach with independent IRBs than with institutionally affiliated boards. “But it could be more widely adopted if some at the institution were ‘in on the plan’ and the results were used for internal quality improvement purposes only,” Newlin suggests.
Protecting participants is “the core purpose of IRB review,” according to Lisa M. Lee, PhD, MA, MS, associate vice president and director of integrity and compliance at Virginia Tech. However, protecting participants does not necessarily mean eliminating all risk involved in research.
Lee says IRBs must assess whether the research risks are reasonable, given the knowledge that will be gained from the study, and whether these risks are minimized to the extent possible. “Most evaluations of IRBs center on how efficient they are,” Lee says.
IRBs consider how quickly they process reviews, or how much back-and-forth they request from researchers. “The harder thing to evaluate is how effective they are, how well they protect research participants,” Lee says.
It is challenging to measure something the IRB prevented. That said, it is possible to examine if IRBs advise researchers appropriately regarding reducing risks. Lee says IRBs can do this by looking at the types of questions and requests the IRB has in response to the risks in a particular research protocol.
Lee suggests one could assess recently approved studies for risks related to the research, and examine how the IRB requested the researcher mitigate those risks. For example, if a study questionnaire asks residents about sensitive topics that could trigger psychological distress (e.g., suicidal thoughts or symptoms of PTSD), the IRB should have ensured the research team was qualified to identify the fact the questions could trigger such a response.
The IRB also should have ensured the team kept a trained professional on hand to intervene immediately or as needed. An evaluation of these types of IRB actions could help determine whether the IRB is meeting their obligation to reduce risk to the extent possible. “While this will not directly answer the question about what we prevented, it will get us a little closer to understanding how IRBs advise and require researchers to reduce the chance that something harmful occurs,” Lee says.
REFERENCES
- Lynch HF, Eriksen W, Clapp JT. “We measure what we can measure”: Struggles in defining and evaluating institutional review board quality. Soc Sci Med 2022;292:114614.
- Kutz G. Undercover tests show the institutional review board system is vulnerable to unethical manipulation. Government Accountability Office. 2009.
It is possible to examine if IRBs advise researchers appropriately regarding reducing risks. Consider the types of questions and requests the IRB has in response to the risks in a particular research protocol.
You have reached your article limit for the month. Subscribe now to access this article plus other member-only content.
- Award-winning Medical Content
- Latest Advances & Development in Medicine
- Unbiased Content