There are three types of mendacity in the world, lies, damn lies and statistics.
– Mark Twain
The Food and Drug Administration (FDA) maintains a large, but incomplete repository of randomized controlled trials (RCT) as part of its new drug application process. Before conducting new drug application trials, pharmaceutical companies must register them with the FDA. This registration includes pre-specifying various outcome measures and analytic methods. Pre-specification is supposed to ensure the integrity of a trial, but there is a catch-22 here. What if the investigators already have an idea-or prior knowledge of some trial results? This is a form of research bias know as HARKing or “hypothesizing after the results are known”, an open invitation for audacious cheating.
Key findings in a recently published combined (meta)-analysis (see (Psychother Psychosom. 2010;79:267) compared all trials for new drugs approved by the FDA from 2001 to 2002 with publication status of these trials 5 years later. The authors found:
New drug application studies with favorable outcomes were almost five times more likely to be published as those with unfavorable ones. 26.5% of pre-specified primary outcome measures were omitted from journal articles of new drug trials. Of the 43 primary measures not supporting efficacy, 20 (47%) were not included in the published results.
The analysis of these articles, in other words, showed significant “publication bias,” HARKing, in other words, a nice way of describing research inflating the apparent effectiveness of new drugs. Is cherry picking the results of drug trials simply a form of lying? Or did I use a dirty word?