Category: Medicine
Sloppyness and effect size correlate linearly in clinical stem cell trials
Discrepancies in the publication of clinical trials of bone marrow stem cell therapy in cardiology scale linearly with effect size! This is the shocking but not so surprising result of a study in BMJ that found over 600 discrepancies in 133 reports from 49 trials. Trials without discrepancies (only 5!) reported neutral results (i.e. no effect of therapy on enhancement of ejection fraction). The most spectacular treatment effects were found in those trials with the highest number of discrepancies (30 and more).
Exploratory and confirmatory preclinical research
In the current issue of PLOS Biology Kimmelman, Mogil, and Dirnagl argue that distinguishing between exploratory and confirmatory preclinical research will improve translation: ‘Preclinical researchers confront two overarching agendas related to drug development: selecting interventions amid a vast field of candidates, and producing rigorous evidence of clinical promise for a small number of interventions. They suggest that each challenge is best met by two different, complementary modes of investigation. In the first (exploratory investigation), researchers should aim at generating robust pathophysiological theories of disease. In the second (confirmatory investigation), researchers should aim at demonstrating strong and reproducible treatment effects in relevant animal models. Each mode entails different study designs, confronts different validity threats, and supports different kinds of inferences. Research policies should seek to disentangle the two modes and leverage their complementarity. In particular, policies should discourage the common use of exploratory studies to support confirmatory inferences, promote a greater volume of confirmatory investigation, and customize design and reporting guidelines for each mode.’
Blogging as a form of postpublication review
In a Neuron View article, Zen Faulkes argues for blogging as a kind of postpublication peer review. He is a veteran blogger, and science tweeter, and knows what he is talking about. The article compares social media to the the classical forms of scientific discourse (from letter to the editor to talk at a conference) and likens science blogging to an online research conference, although which a much wider reach, even into the lay community. Read it here: Faulkes Neuron View
Systemic flaws of biomedical research ecosystem
In the current issue of the Proceedings of the National Academy of Science (USA), four heavyweights, Bruce Alberts, Marc W. Kirschner, Shirley Tilghman, and Harold Varmus, provide fundamental criticism of the US biomedical research system, and offer ideas for ‘Rescuing US biomedical research from its systemic flaws’. Their main point is that ‘The long-held but erroneous assumption of never-ending rapid growth in biomedical science has created an unsustainable hypercompetitive system that is discouraging even the most outstanding prospective students from entering our profession—and making it difficult for seasoned investigators to produce their best work. This is a recipe for long-term decline, and the problems cannot be solved with simplistic approaches.’ Most of the issues they raise are equally applicable to European biomedical research. Full article: PNAS-2014-Alberts-5773-7
Due dilligence in ALS research
Steve Perrin in a recent issue of NATURE (Vol 507, p.423) summarizes the struggle of the amyotrophic lateral sclerosis (ALS) field to explain the multiple failures of clinical trials testing compounds to improve the symptoms and survival of patients with this disease. He reports the efforts of the ALS Therapy Development Institute (TDI) in Cambridge, Massachusetts, to reproduce the results of around 100 mouse studies which had yielded promising results. As it turned out, most of them, including the ones that led to clinical trials, could not be reproduced, and those where an effect was seen it was dramatically lower than the one reported initially. He discusses a number of measures that need to be taken to improve this situation, all of which have been emphasized independently in other fields of biomedicine where bench to bedside translation has failed.
Have the ARRIVE guidelines been implemented?
Research on animals generally lacks transparent reporting of study design and implementation, as well as results. As a consequene of poor reporting, we are facing problems in replicating published findings, publication of underpowered studies and excessive false positives or false negatives, publication bias, and as a result difficulties in translating promising preclinical results into effective therapies for human disease. To improve the situation, in 2010 the ARRIVE guidelines for the reporting of animal research (www.nc3rs.org.uk/ARRIVEpdf) were formulated, which were adopted by over 300 scientifc journals, including the Journal of Cerebral Blood Flow and Metabolism (www.nature.com/jcbfm). Four years after, Baker et al. ( PLoS Biol 12(1): e1001756. doi:10.1371/journal.pbio.1001756) have systematically investigated the effect of the implementation of the ARRIVE guidelines on reporting of in vivo research, with a particular focus on the multiple sclerosis field. The results are highly disappointing:
‘86%–87% of experimental articles do not give any indication that the animals in the study were properly randomized, and 95% do not demonstrate that their study had a sample size sufficient to detect an effect of the treatment were there to be one. Moreover, they show that 13% of studies of rodents with experimental autoimmune encephalomyelitis (an animal model of multiple sclerosis) failed to report any statistical analyses at all, and 55% included inappropriate statistics.. And while you might expect that publications in ‘‘higher ranked’’ journals would have better reporting and a more rigorous methodology, Baker et al. reveal that higher ranked journals (with an impact factor greater than ten) are twice as likely to report either no or inappropriate statistics’ (Editorial by Eisen et al., PLoS Biol 12(1): e1001757. doi:10.1371/journal.pbio.1001757).
It is highly likely that other fields in biomedicine have a similar dismal record. Clearly, there is a need for journal editors and publishers to enforce the ARRIVE guidelines and to monitor its implementation!
Found in translation
Lost or found in translation? Stroke is a major cause of global morbidity and mortality, yet therapeutic options are very limited. Numerous preclinical studies promised highly effective novel treatments, none of which have made it into practice despite a plethora of clinical trials. This failure to bridge the gap between bench and bedside deeply frustrates researchers, clinicians, the pharmaceutical industry, and patients. Dirnagl and Endres argue that despite the apparent translational failures in neuroprotection research, and counter to current nihilism, basic and preclinical stroke research has in fact been able to predict human pathophysiology, clinical phenotypes, and therapeutic outcomes. The understanding of stroke pathobiology that has been achieved through basic research has led to changes in stroke care whose value can be demonstrated. Preclinical investigations have informed the clinical realm even in the absence of intermediary phase 2 or phase 3 trials. Their arguments rest on examples of successful bench-to-bedside translation in which experimental studies preceded human trials and successfully predicted outcomes or phenotypes, as well as on examples of successful ‘back-translation’, where studies in animals recapitulated what we already knew to be true in human beings. An analysis of the reasons for the apparent (or only perceived) translational failures further strenghtens their proposition, and suggests measures to improve the positive predictive value of preclinical stroke research. Researchers, funding agencies, academic institutions, publishers, and professional societies should work together to harness the tremendous potential of basic and preclinical research, in stroke research as well as in other fields of medicine
Ulrich Dirnagl and Matthias Endres. Found in Translation: Preclinical Stroke Research Predicts Human Pathophysiology, Clinical Phenotypes, and Therapeutic Outcomes. Stroke. 2014; 45: 1510-1518
Nachkochen unmöglich
Das Heft 3 des Laborjournals enthält einen sehr brauchbaren Artikel zur (nicht-) Reproduzierbarkeit von Ergebnissen. Sorry, in German only….
METRICS
The Economist reported that John Ioannidis, together with Steven Goodman, later this month will open the Meta – Research Innovation Center at Standford University (METRICS). Generously supported by the Buck foundation , it will fight bad science, bias, and lack of evidence in all areas of biomedicine. The institute’s moto is to ‘Identify and minimise persistent threats to medical research quality’. Those who have followed the work of Ioannidis and Goodman know that this is good news indeed! A concise overview of Ioannidis research can be found in this online article at Maclean’s.
The probability of replicating ‘true’ findings is low…
Due to small group sizes and presence of substantial bias experimental medicine produces a large number of false positive results (see previous post). It has been claimed that 50 – 90 % of all results may be false (see previous post). In support of these claims is the staggerlingly low number of experiments that can be replicated. But what are the chances to reproduce a finding that is actually true?
To infinity, and beyond! 