An emerging body of research reveals that past studies — some of which may form the basis of current policies and recommendations — cannot be replicated by investigators today. This lack of “reproducible” research may undermine current studies based on prior findings, particularly as investigators look at the risk-benefit ratio for people participating in a clinical trial, says Barbara K. Redman, PhD, MBE, of the division of medical ethics at New York University Langone Medical Center.
“Having served on an IRB myself for three years, if you are looking at the risk-benefit ratio, you are assuming that the studies on which they are basing it are valid,” she says. “Now we are finding they might not be. A percentage of them might not be, and that might vary by field. I think this strains IRBs in trying to fulfill their requirements.”
The lead author of a recently published paper1 on the issue, Redman cites studies that estimate anywhere from 22% to almost half of published results in biomedical research actually can be validated.2,3
“[Reproducibility] varies across fields, but let me just stop and say when scientists are working at the edge of new innovations or new fields, frequently things are not reproducible,” she says. “That is just part of the process, as you might well imagine, and it is a normal process to some extent to have non-reproducibility.”
Thus, it is more of an issue of degree than kind, as the large proportion of studies that lack reproducibility has underscored the scale of the problem even to those who first questioned such findings.
“It is contested, though I don’t see as many people contesting it now,” Redman says. “I think they are shocked, to tell you the truth. This particularly has been a political issue because pharmaceutical companies depend on academic and other research to get a basic idea of [product efficacy], but when [subsequent researchers] go back and try to reproduce those studies, they can’t.”
IRBs should “require that research protocols contain explicit probability statements about likely risks and benefits, based on a comprehensive review of prior studies and meta-analyses addressing reproducibility … Such estimates are essential for IRB judgment about minimizing risk, for determining an appropriate risk-benefit ratio for presentation as a part of the informed consent process, and in seeking to facilitate the informed choices of potential research subjects,” Redman wrote in the paper.
For their part, investigators should include assessments of past studies in terms of reproducibility and perceived risk. As such requirements are hardwired into IRB and investigator protocols and policies, more research should become reproducible, she notes. As it stands, unreproducible research may pose a risk to research participants, particularly those from vulnerable populations.
As defined in the paper, reproducibility is an “umbrella term” that includes whether an original study can be repeated and yield the same results. Other nuances to the concept are “replicability” and “validation,” but all terms generally reflect the enduring quality of data to provide a jumping-off point for subsequent research.
“We are trying to dig into all of the reasons that something might not be reproducible, and those factors vary dramatically,” Redman says.
Among the common reasons cited are unrecognized study variables, poor study design, inadequate documentation of findings, and overstatement of the benefits of the research.
“Lack of reproducibility by independent investigators may signal that research misconduct took place and that, as a result of fraudulent data, current research participants and subsequent patients could be harmed if research and medical practices are based on such data,” Redman wrote.
In addition to ethical issues, there are now regulatory incentives to address reproducibility, she says.
Though the study was published before it was issued, recent final guidelines by the Department of Health and Human Services on clinical trials emphasize that, “the public availability of results information helps investigators design trials and IRBs review proposed trials, by allowing them to weigh the proposed study’s risks and benefits against a more complete evidence base than is currently available through the scientific literature.”4
Redman sees that as a strong endorsement of ensuring reproducibility in clinical trials, though she concedes IRBs may not currently be geared up to address the issue.
“That is the crux of it,” Redman says. “These regulations will go into effect in January 2017 and be enforced three months thereafter. I think there is no question that it will be a catch-up situation for IRBs. The first thing to do is to be sure that all trials are registered. [Then] an IRB could look on ClinicalTrials.gov for a particular kind of proposal, see all of the studies that have been done, and figure out whether they have been reproduced. Some people say this is more than an IRB ought to do. Our response is that it is central to their ethical role, which is to control harms and to figure out if the risk-benefit ratio is OK.”
Bottom-line protections for research subjects essentially assume that a new study is based on valid, reproducible research — which, unfortunately, may not be the case. As the issue is addressed and rectified, IRBs must make good faith efforts to do what they can to seek evidence of reproducibility and point out when it is lacking.
“We understand that IRBs may have difficulty fulfilling this right now, but they need to work on it,” Redman says.
REFERENCES
- Redman BK, Caplan AL. Limited Reproducibility of Research Findings: Implications for the Welfare of Research Participants and Considerations for Institutional Review Boards. IRB: Ethics & Human Research 2016 July-August 2016;8-10.
- Ioannidis J. How not to be wrong. New Scientist 2014;22:32-33.
- Ioannidis J. Improving validation practices in “omics” research. Science 2011;334:1230-1232.
- HHS. 42 CFR Part 11. Clinical Trials Registration and Results Information Submission; Final Rule. Fed Reg 2016; 81:(183):64,982-65,157.
An emerging body of research reveals that past studies — some of which may form the basis of current policies and recommendations — cannot be replicated by investigators today.
You have reached your article limit for the month. Subscribe now to access this article plus other member-only content.
- Award-winning Medical Content
- Latest Advances & Development in Medicine
- Unbiased Content