An important study was published this week in the JAMA, “Persistence of Contradicted Claims in the Literature.” In it, the authors conclude that “Claims from highly cited observational studies persist and continue to be supported in the medical literature despite strong contradictory evidence from randomized trials.”*
The authors examined various trials on the effectiveness of Vitamin E for cardiovascular disease prevention, beta-carotene for cancer prevention and estrogen for prevention of Alzheimer’s. They disclosed various biases, adding that genuine diversity of opinion in randomized trials may “lead to a decrease in the absolute frequency of citations…” and a considerable delay to cite contradicted articles long after the published contradictions. They described an interesting bias seen in scientists as well as non-scientists, who may be vulnerable to shared beliefs influencing the interpretation of scientific data, the so-called “wish bias.”
These observations have long been discussed in older medical literature. Odds ratios, as in “this horse is favored 4 to 1” and “P values”, examples of relative differences, are a favorite ploy of drug advertisers and clinical researchers publishing in medical journals. In an important British Medical Journal study the significance of P values was compared with observational data in 260 abstracts of randomised controlled trials in PubMed, and concluded that “Significant results in abstracts are common but should generally be disbelieved.”
Another study examined outcome reporting bias in 519 trials with 10,557 outcomes. The authors concluded that “Incomplete reporting of outcomes within published articles of randomised trials is common and is associated with statistical non-significance. The medical literature therefore represents a selective and biased subset of study outcomes…”
Read this and weep. Better still, maintain a healthy skepticism when the news anchor or TV doctor tells you about the latest blockbuster treatment for cancer, heart disease, or unsightly skin.
* “A clinical trial is an experiment conducted with patients as subjects. … The strongest experimental design …is the randomized design in which patients are randomly assigned to treatment groups. An important distinction in the purpose of clinical trials is that between therapeutic trials (comparing treatment methods) and prophylactic (or prevention) trials.”