Data driven: Accreditation group releases metrics for IRB performance
Encouraging `evidence-based' human subjects protection
Using information compiled from its clients, the Association for the Accreditation of Human Research Protection Programs (AAHRPP) is providing an intriguing snapshot of IRB operations.
The data, collected from nearly 200 institutions during 2009, shows the kinds of protocols IRBs are encountering, their workloads, review times and staffing and funding. The goal, says AAHRPP President Marjorie Speers, PhD, was to get a sense of what the performance of human research protection programs currently looks like so that institutions can start examining whether they're functioning effectively.
"We undertook this project because we realized there's very little information that exists about IRB performance or performance of human research protection programs," Speers says. "Collecting information is integral to improving quality."
"There has been much discussion lately as part of health care reform about evidence-based medicine. We want to encourage evidence-based human subjects protection."
The information used in the metrics was gathered during AAHRPP's application and renewal processes.
While the accrediting organization's client list isn't completely representative of the IRB community at large there are few community hospitals among the group, for example Speers believes the results are generalizable to institutions outside AAHRPP.
And the data can give some hints as to the direction in which human subjects protection is evolving. One example concerns the regulatory agencies that organizations report having oversight over them.
While 86% reported following regulations and guidance from the U.S. Department of Health and Human Services (through the Office for Human Research Protections), 99% reported following Food and Drug Administration regulations and guidance.
"We're interpreting this to mean that industry-sponsored research is really driving the research enterprise," Speers says. "When we think about what regulatory bodies have the greatest influence over research organizations, it's now the FDA and I think that's a real change."
Workloads, use of external IRBs
Asked about specific types of research that can be ethically complex, nearly all institutions said they conduct some research with vulnerable populations 95% conduct research with adults with diminished capacity, 76% with pregnant women, and 71% with children.
"We were surprised the numbers there are as high as they are," Speers says.
The numbers don't necessarily mean that IRBs are conducting drug trials with those vulnerable subjects, however, says Mark Schreiner, MD, chairman of the Committee for the Protection of Human Subjects at The Children's Hospital of Philadelphia.
"There could be a lot of research with pregnant women, but it could be surveys or questionnaires," Schreiner says. "I don't know whether it includes the kind of research we'd like to go on with pregnant women, namely studies of the safety and efficacy of drugs."
While AAHRPP does include some large research institutions, as Speers noted, most are smaller organizations. So while the average number of new and continuing protocols overseen by institutions during 2009 was 851, the median number was a much smaller 306.5. Most of those protocols received either expedited or full review IRBs reported exempting an average of 139 protocols in 2009 (the median was 26.5).
Of the institutions that have their own IRBs, 53% also rely on at least one other IRB for some studies this could include formal arrangements with an affiliated IRB, a central IRB or an independent IRB. Speers notes that out of that group, nearly three-quarters use an external IRB for less than 10% of their studies.
She says AAHRPP hopes to see the reliance on outside IRBs increase in future years, particularly for multisite studies, as institutions become more comfortable with the idea. Speers says one value of accreditation is that as AAHRPP's standards become more widely known and accepted, it may lead institutions to be more comfortable ceding review to another accredited institution.
Sixty-five percent of organizations reported compensating their IRB chairs, while 42% compensate vice chairs, 34% do so for affiliated IRB members and 57% for nonaffiliated members.
Compensation may take different forms, from direct payments to members to payments to their departments or releasing them from other duties.
"We didn't collect the method of payment it's an area we want to collect more information on in the future," Speers says.
Schreiner says he'd like to see more information about what percentage of IRB members' time is compensated.
"I'm compensated and my vice chair is compensated for basically all of our time," he says. "But at some institutions, people might get paid for 20% of their time."
Review times – what can they tell us?
The length of time it takes to conduct IRB review has been a point of discussion in the research community. Investigators often blame IRBs for delays in initiating research.
And it's difficult to define what constitutes a "good" review time. Are those who take less time doing a thorough job? Are those who take more time operating efficiently?
While Speers AAHRPP's report doesn't definitively answer those questions, she says it does provide valuable input to the discussion:
- The average time for a protocol to take from submission to review by a convened IRB was 25 days; the average from submission to final approval by a convened IRB was nearly 49 days.
- Expedited and exempt studies proceeded more speedily average time from submission to final approval for expedited studies was nearly 30 days, and from submission to an exempt determination was about 18 days.
"Is 25 days (from submission to review by an IRB) good or bad? We can't say yet, we don't know," Speers says. "The expedited review is taking about 18 days – could that be shorter? Maybe."
Organizations can look at these numbers and see whether they're falling outside the averages and whether that might suggest some room for improvement, she notes.
But Schreiner says that without further details, the review times, and other metrics from the AAHRPP report, would be difficult to use for benchmarking by individual institutions. With review times, for example, much of the time between submission of a protocol and final approval is spent waiting for an investigator to respond to the IRB's questions or comments.
"I would want to benchmark myself by how long it takes from the time something is submitted until it gets distributed to an analyst for pre-review," he says. "How long before it goes from the analyst to a committee? How long from a committee to a letter? How long after that letter does it then take for the investigator to respond? I'm interested in the internal processes within the IRB that take time."
And he says that other results, including numbers and types of protocols reported, would vary widely by size and type of institution.
"I think that this is a great first step," Schreiner says of the AAHRPP report. "What we need now is to be able to drill down further so we could compare ourselves to (other children's hospitals), organizations like ours doing similar research."
Denise Roe, MSM, CCRP, CIP, RAC, director of the Human Research Protection Program at Vanderbilt University in Nashville, TN, is pleased with the data AAHRPP released. Too often, when someone attempts to pull together information from IRBs, it's with a specific question or hypothesis in mind, which often skews the way questions are presented, she observes.
"With AAHRPP, they weren't trying to answer a question, and I think that's why it makes a difference," Roe says. "AAHRPP isn't saying, 'We think we know a better way.' What they say is 'What is the way that you do it? How is it being done?'"
She said reading the report, gave her ideas for data her institution might track more carefully.
"I actually enjoyed reading their report. It made me think, 'Oh, we should collect that,' or 'That might be another way to look at it.'"
Roe says trying to elicit a lot more detail from the AHRPP client group won't necessarily provide a clear comparison between organizations.
"Because we're not all the same anyway, comprehensive metrics are difficult," she says. "I think if you try to get too specific, you are forming your concept of what it should look like from where you're sitting."
Speers says response to the report so far has been positive. She's already getting requests from clients to collect additional data in future years.
"This is an activity that needs to be driven by the research enterprise," she says. "Then it's going to be valuable to research institutions, to IRBs, to investigators and the public."
To view the AAHRPP metrics, visit the organization's website at www.aahrpp.org
Using information compiled from its clients, the Association for the Accreditation of Human Research Protection Programs (AAHRPP) is providing an intriguing snapshot of IRB operations.You have reached your article limit for the month. Subscribe now to access this article plus other member-only content.
- Award-winning Medical Content
- Latest Advances & Development in Medicine
- Unbiased Content