Product efficacy: Evaluating the evidence
March 1, 1999
Product Pointers
Product efficacy: Evaluating the evidence
By Liza G. Ovington, PhD, CWS
President
Ovington & Associates
Ft. Lauderdale, FL
Products for wound prevention and treatment are more abundant — and, perhaps, more redundant — than ever before.
For example, the Ostomy Wound Management Buyer’s Guide issue for 1998 lists 22 film dressings, 22 alginate dressings, 42 hydrocolloid dressings, 42 seating surfaces, and 62 nonpowered mattresses. Health care providers are exposed to new products at professional meetings, in newsletters and journals, on the Internet, by a colleague, or by a manufacturer’s sales representative. This multitude of products vies for attention with clever advertising campaigns, T-shirts, buttons, and even plush toys.
But stuffed animals aside, how do you determine if you should use the new product? You need data. The manufacturers know the trinkets only get your attention for a moment, and in that moment they have to give you their best evidence of clinical efficacy in a pertinent patient population. So you walk away not only with a stuffed animal but with a few pounds of paper, too. In order to wade through that paper, or even stroll down the aisles of a poster session, you need to know a thing or two about clinical research, the various forms it can take, and how the results stand up to scrutiny.
Clinical studies fall into two broad categories: those that are simply observational or descriptive, and those that are experimental or comparative. Another way to refer to these two categories is uncontrolled and controlled, respectively. Within each of the two categories, there are subcategories, each with its own set of pluses and minuses. With any clinical study type, there are two major types of mistakes or error that can be made: random error or non-random error.
Random error has been defined as the difference between the estimated effect of a treatment and the true effect of the treatment (if it were known) that is due only to chance (Margolis, 1998). The magnitude of this type of error is often expressed as a p-value. Non-random error (also called bias) has been defined as the difference between the observed effect of a treatment and the true effect that is due to a systematic flaw. The most common areas of bias or systematic flaws in clinical studies are in patient selection, data collection, or overlooking a variable that could have a confusing effect on the study. Let’s briefly discuss some of the more commonly encountered types of clinical research in wound healing and their potential for error or bias:
• Case study or case series.
This type of descriptive study usually looks at a single treatment in a single patient or series of patients. They are uncontrolled, meaning they are not comparing one treatment to another directly. Case studies are popular because they are quick to do and make a great poster. A case study or series may get your interest because everyone wants to see how a product performs in a human patient as opposed to an animal or cells in a petri dish, but it cannot establish efficacy of a product. One reason why is the placebo effect.
A placebo is an inert substance used as a control in some experiments. The placebo effect is the measurable or observable effect on a person or group that has been given a placebo rather than an active substance. H. K. Beecher evaluated more than two dozen studies and calculated that 32.5%, or about one-third, of those in the studies improved due to the placebo effect (Beecher, 1955). Other studies calculate the placebo effect as being even greater than Beecher claimed. The placebo effect and its potential magnitude should make it evident that the use of uncontrolled studies to infer the effectiveness of any treatment, therapy, or program should be taken with a grain of salt. A controlled study must be done to determine whether any improvement is actually due to the treatment rather than the placebo effect or some other variable.
Many who have done clinical research have noted that the process of being in a study can affect patient compliance, as well as engaging the other participants to be more aware and engaged in overall care. One thing that case studies can do is stimulate ideas or garner interest for a more rigorous trial.
• Cross-sectional studies.
Also referred to as prevalence studies, cross-sectional studies are descriptive or observational studies that examine treatment status or outcome at one specific point in time. Prevalence studies often are done to determine an initial state prior to and after an intervention. A common example is when facilities do a pressure ulcer prevalence study prior to instituting a new type of support surface, wound care algorithm, or skin care products. After a period of time, they repeat the study to determine any change in in-house pressure ulcer development or prevalence. A change in outcome may be due to the intervention, but it is difficult to rigorously prove a temporal relationship between intervention and outcome. Cross-sectional studies are not sufficient to establish treatment efficacy.
• Case-control studies.
Case-control studies also are called retrospective studies. In this type of research, the treatment history and outcomes of a group of subjects with the condition of interest (cases) are compared to that of a group of subjects without the condition (controls). For example, patients with healed wounds or wounds treated with a new type of dressing are compared to those with nonhealed wounds or wounds treated with a standard type of dressing. Patient group selection bias is a potential flaw in this type of study, so selection parameters must be carefully defined and matched for both cases and controls. Individual patient selection for treatment is another source of bias and can rarely be corrected since it is usually not known in a retrospective review. Information bias also is possible because the retrospective review of information may reveal poor documentation practices or inconsistencies in measuring or recording data. Retrospective studies are not often used to establish treatment efficacy, but have been used to supply evidence for rare outcomes (e.g., retrospective reviews of incidence of wound infection under occlusion) or outcomes that occur over long periods of time.
• Randomized controlled trials.
The randomized controlled trial (RCT) is often considered the gold standard for clinical research. The salient features of the RCT are that it is comparative (i.e., includes a control or placebo) and the patient selection method is not biased by the investigator because it is randomized by some method other than human decision. However, the RCT is not always the most appropriate or feasible type of study for determining a particular outcome. For example, if you want to know about risk factors, a case-control study may be a better design. If you want to know something about prevalence or the accuracy of a particular diagnostic test, then a cross-sectional study may be better. It also is possible to run into ethical issues in performing a RCT.
The fact that a study is randomized and controlled does not necessarily mean it is a great study. The method of randomization is a very important factor. Randomization of patients is critical to balance any unknown and therefore unaccounted-for factors that could influence outcomes. It is important for every patient in the study to have an equal chance of being assigned to the treatment or control group. The best randomization method is one that uses a mathematical technique such as a random numbers table to assign patients. When methods such as days of the week, medical record numbers, or a coin toss are used to assign patients to test or control, the study is not considered truly randomized.
Ensure evaluator of outcomes is blinded’
RCTs usually are blinded or double-blinded. Some studies are straightforward to blind. A pill or cream or gel may either contain active ingredients or not. Electrical stimulation may have current flowing or not. In dressing studies, it often is difficult or impossible to blind either the patient or the investigator to the identity of the test dressing vs. the control dressing. For instance, a piece of gauze looks quite different from a foam or film. In this case, it is important that the person who evaluates the wounds (i.e., measures them or rates them in some way) be blinded to the treatment by only being brought in after the dressings have been removed.
This is not an exhaustive discussion of the types of clinical research possible, but the objective has been to touch on the most common studies in the wound-healing literature and to point out some differences. In general, an observational (uncontrolled) type of study generates information but cannot be taken as strong evidence of comparative efficacy. A controlled study is a bit more rigorous and minimizes the placebo effect, but random patient assignment and blinding of the evaluators are vital to balance sources of bias.
References and Further Reading
1. Roma A, Bolton L, McNally A. "Controlled Clinical Evaluations Versus Case Studies: Why Wound Care Professionals Need to Know the Difference." In: Krasner D, Kane D, eds. Chronic Wound Care. 2nd ed. Wayne, PA: Health Management Publications; 1997, pp. 373-382.
2. Margolis D, Baumgarten M. Basic epidemiology: wound treatment study designs. Wounds 1998; 10 (Suppl. D):1D-5D.
3. Beecher HK. The powerful placebo. JADA 1955; 159: 1,602-1,606.
You have reached your article limit for the month. Subscribe now to access this article plus other member-only content.
- Award-winning Medical Content
- Latest Advances & Development in Medicine
- Unbiased Content