In 2009, Chalmers and Glasziou investigated sources of avoidable waste in biomedical research and estimated that its cumulative effect was that about 85% of research investment is wasted (Lancet 2009; 374: 86–89). Critical voices have since then questioned the exceedingly high number (85%), or claimed that because of non-linearity’s and idiosyncrasies of the biomedical research process a large number of failures are needed to produce a comparably small number of breakthroughs, and therefore hailed the remaining 15%. Waste is defined as ‘resources consumed by inefficient or non-essential activities’. Does progress really thrive on waste?

It is of course absurd to claim that spoilage of resources could foster progress. Rather, the argument that a 85/15 ratio of waste to productive output isn’t bad after all is based on confusing ‘waste’ with ‘failure (or attrition)’. Indeed, not only has it been argued that a large part of biomedical research is waste, currently almost every medical field is frustrated by the very low success rates of bench to bedside translation and attrition of drug development in late clinical stages. In many fields, most drugs which proved highly effective in preclinical modelling failed to benefit patients, and only an exceedingly small portion of drugs entering clinical testing proceed to licensing, let alone affect major improvements over current medical practice. Is this high rate of attrition wasteful?

It all depends! As long as ‘negative’ (or even false positive!) preclinical or clinical research is of high quality, and the hypotheses tested sound, it may not be wasteful at all! Preclinical exploratory research (see previous post) operates in an almost unlimited space of pathophysiological theories, hypotheses, and complex approaches, most of which are unbeknownst to the scientists apriori not true or informative. Due to these low base rates, and small sample sizes (as testing of many hypotheses limits resources), false positives abound. In addition, effect sizes are necessarily inflated in small, low-powered studies that can only detect effects that happen to be large (“Winner’s curse”, Button et al.). On replication, in particular with increased power, such exaggerated effect sizes will recede (regression to the mean). As long as we are aware of this, and bias is minimized and internal validity maintained high, all this is acceptable to a certain degree! Following exploration, the most promising treatment strategies should then be further validated and confirmed, using sufficiently large sample sizes to reduce false positive as well as false negative results (i.e. Type I and II errors). In addition to high internal validity, confirmatory preclinical research needs a strong focus on external validity to be able to generalize the results to human populations, settings, and treatment variables. Like it or not, clinical testing must similarly lead to ‘negative’ results, as well as false negatives and false positives. In particular phase II randomized controlled clinical trials explore various indications, dosages, treatment schemes, etc. In this context negative results are important as they establish the boundaries for effectiveness of a drug, and are therefore necessary costs of the drug development process.
As a consequence, the current mantra of reducing the attrition rate (ideally to 0!) and speeding up translation, although understandable, is to a certain degree based on a misconception of the biomedical research process. In the long run, this misconception may even lead to increases in the number of large scale late stage failures, and expose patients to ineffective or even unsafe drugs. Just as with preclinical research: ‘Negative’ results which are based on high quality studies are as informative and relevant as ‘positive’ ones. And not publishing them or preventing access to their results, therefore is a ‘sin’. It is all about evidence, not primarily about success!
So does progress thrive on waste? Of course not, but it may thrive on attrition, or, as John Ioannidis puts it in the title of a recent commentary: ‘Translational research may be most successful when it fails.’ In the words of Jonathan Kimmelman and Alex John London: ‘Many discussions of clinical translation and its pathologies presume that its main output is hardware: new drugs, vaccines, devices, and diagnostics. We disagree. We argue that the principal output of clinical translation is information — in particular, information about the coordinated set of materials, practices, and constraints needed to safely unlock the therapeutic or preventive activities of drugs, biologics, and diagnostics” (Hastings Center Report March-April 2015). The point here is: Waste abounds in biomedical research: as a result of inappropriate research design, methods, and analysis; lack of access to results or biased reporting; inefficient research regulation, etc., and we should fight it. We should not, however, confuse ‘waste’ with negative results or attrition, which are necessary elements of a rational research process.