Clinical results too good to be true?
Most likely they are
Clinical trials that show a treatment effect early on face a dilemma. If the effect is real, one ethical argument goes, then it is imperative to give all involved access to the drug as quickly as possible, starting with the placebo group.
The problem is that efficacy early in a clinical trial may not be real, but instead a mere statistical fluke. For that reason, Gordon Guyatt, professor of clinical epidemiology and biostatistics and medicine at McMaster University in Hamilton, Ontario, calls that ethical argument "honest but misguided."
In the Nov. 2 issue of the Journal of the American Medical Association, Guyatt and his colleagues cast a critical eye on clinical trials ended early. While the words describing such trials are often nothing short of bombastic — an editorial that accompanies the JAMA paper has one researcher saying it was "vital to tell the world immediately" of their results — the first look often doesn’t hold up.
Because in clinical trials, it’s a number, not a picture, that is worth a thousand words. And according to the authors’ analysis, the numbers paint a sobering picture.
Ninety percent of 143 trials that were stopped early for benefit ended up failing to provide a full interim statistical analysis of the truncated trial, and only about two-thirds of them provided a statistical justification for their decision at all. In the remaining third, in the words of the authors, "a statistical approach to monitoring the trial was either not used or not specified in the report."
To evaluate whether the decision to stop early is statistically sound, or whether it is even possible to evaluate whether the decision is sound, the authors, who are from McMaster University, Ontario; the Mayo Clinic College of Medicine in Rochester, Minn.; the University of Toronto; Basel University Hospital in Switzerland; the University of Buffalo in New York; and the Italian National Cancer Institute, Rome, analyzed such trials for three pieces of information: the planned sample size, the interim analysis that led to the decision to stop a trial early and the statistical rule used for the decision to stop a trial early. Fewer than half of the trials reported all three numbers.
Most troubling was one particular inverse correlation: The larger an observed effect, the less data it was based on. That is, the more of a chance that it was due to a statistical phenomenon called, fittingly, the random high.
Since dramatic findings are the ones most likely to be published in high-impact journals and influence clinical practice, it is problematic that many of those dramatic findings turn out to be statistical flukes.
The authors described typical truncated trials as being disproportionately industry-funded drug trials in cardiology, cancer, and HIV.
Guyatt explained that the reason serious diseases are disproportionately represented is that clinical trials are most often stopped for benefit when there is the perception that doing so will save lives: "If you’re decreasing runny nose, that’s a much less compelling reason," he said.
The authors also noted that a majority of those trials were published in only five high-impact general medical journals, including JAMA.
For clinicians, the authors recommend a healthy dose of skepticism when evaluating reports of large benefits in clinical trials, including cross-checking with what is already known about a drug. They cited an example of a clinical trial that was stopped early for benefit despite results that were inconsistent with the researchers’ expectations, with earlier clinical trials, and with clinical practice, calling the data "likely too good to be true."
For those running clinical trials, Guyatt has an equally blunt message: "If you can avoid it, don’t stop early under any circumstances for benefit. If you are forced to have a stopping rule" — which grant reviewers sometimes will insist on — "choose an extremely conservative rule and don’t stop until as late in the trial as possible."
Clinical trials that show a treatment effect early on face a dilemma. If the effect is real, one ethical argument goes, then it is imperative to give all involved access to the drug as quickly as possible, starting with the placebo group.You have reached your article limit for the month. Subscribe now to access this article plus other member-only content.
- Award-winning Medical Content
- Latest Advances & Development in Medicine
- Unbiased Content