Tall, white-haired, and wearing thick glasses, Hoffman looks the part of an elder statesman of medicine. He says he became interested in the interpretation of medical literature when he was just starting out as a resident physician at UCLA. He read studies and their interpretations voraciously, and eventually other physicians began coming to his talks to residents and medical students on how to interpret the medical literature. “Some studies just didn’t make sense to me,” he says. “I was reading all these things that came to opposite conclusions. They couldn’t all be right.” Besides, Hoffman says, “there were studies that didn’t represent what I was seeing in clinical practice.”
When the NINDS study was published in December 1995, Hoffman paid attention. “It was a big deal,” he says. “If tPA worked, it would be a real advance over what we could offer patients during an acute stroke.” But, he says, “you [should] never believe one study—especially of a drug that has only a small benefit, and especially a study that is contradicted by other studies, as was the case here.” Hoffman is also critical of a study that Genentech says supports the NINDS trial findings. He contends that this study, known as SITS-MOST, shows “how study design and spin can inflate perceived benefit.” This is because “no patient with a severe stroke was allowed into SITS-MOST—by design—so the patients who did get included were virtually certain to do well as a group, no matter what treatment they did or didn’t get. Comparing them to the much sicker patients in the big trials isn’t like comparing apples with oranges—it’s comparing apples with elephants.”
One way to make drugs look better or safer is to report only successful studies while ignoring those with bad results.
Hoffman says that the truth in any drug study can be camouflaged by how it is reported. “One way that’s been done—for many treatments, and not just [clot-busters]—is to use combination end points.” Here’s how it works: A single drug can be tested for a variety of outcomes; for example, a cholesterol-lowering drug can be tested for its effect on cholesterol level, blood pressure, and/or rates of heart failure, heart attack, or death. By combining two or more of these outcomes to create a single category, you can say it helped “A and B” even if it only helped A and not B. For example, although there was no statistically significant effect from tPA in the NINDS trial on the number of patients who died, there was a small decrease in disability for those who survived. With the two factors combined, there was technically a decrease in the combination end point of “death and disability.” From there, it’s a short step to the incorrect assumption that death and disability were each decreased—an assumption made by many physicians and patients.
David L. Brown, chief of the division of cardiovascular medicine at the State University of New York, Stony Brook School of Medicine, calls the use of combination end points a “brilliant marketing tool.” Brown says that while combination end points have a legitimate purpose when researchers are testing drugs for a rare or infrequent outcome, far too often researchers use the information in ways that “mislead both doctors and the general public.”
Another way to make drugs look better and safer than they are is to report or cite only successful studies while ignoring those with bad outcomes. The problem of cherry-picking studies is a very real one, especially for antidepressants, says Erick Turner, a former FDA reviewer, now an assistant professor of psychiatry at the Oregon Health & Science University. Turner recently published research in NEJM showing that “when studies of antidepressants were negative, they were reported as negative only 8 percent of the time—but when studies were positive, they were reported as positive 97 percent of the time.”
Genentech acknowledges that no controlled study has ever shown—or been conducted to show—that tPA “saves lives” in cases of acute stroke. What the NINDS study showed, Genentech spokesperson Krysta Pellegrino says, is that “patients were at least 30 percent more likely to have a decrease in stroke-related disability three months after treatment compared with placebo.” Although Genentech admits that high-risk patients were excluded from analysis in SITS-MOST, Pellegrino says the company believes the data from that study “add to the body of evidence that supports the conclusion that tPA is safe and effective for the treatment of acute stroke.”
