Category: Neuroscience
Open evaluation of scientific papers
Scientific publishing should be based on open access, and open evaluation. While open access is on its way, open evaluation (OE) is still controversial and only slowly seeping into the the system. Kriegeskorte, Walther, and Deca have edited a whole issue on Frontiers in Computational Neuroscience devoted to this topic, with some very scholarly and thoughtful discussions on the pros and cons of OE. I highly recommend the editorial (An emerging consensus for open evaluation), which tries to synthesize the arguments into ’18 visions’. The beauty of their blueprint for the future of scientific publication (which was already published a year ago) is that it is possible to start with the current system and slowly evolve it into a full blown OE system, while checking on the way whether the different measures deliver their promises.
How Science goes wrong
Scepticism regarding the quality and predictiveness of modern science has finally arrived in the lay press. This week The Economist has devoted its issue, including, cover, editorial, and leader to what they call ‘unreliable research’. Even closer to home, this weeks New Scientist (also with cover, editorial and leader) turns on neuroscience, with a similar message and material, and the bottom line that ‘the vast majority of brain research is now drowning in uncertainty.’ A clear signal that it is either time to abandon ship, or to clean up the mess!
Too good to be true: Excess significance in experimental neuroscience
In a massive metaanalysis of animal studies of six neurological diseases (EAE/MS; Parkinsons; Ischemic stroke; Spinal cord injury; Intracerebral hemorraghe; Alzheimer’s disease) Tsilidis at al. have demonstrated that the published literature in these fields has an excess of statistically significant results that are due to biases in reporting (PLoS Biol. 2013 Jul;11(7):e1001609). By including more than 4000 datasets (from more than 1000 individual studies!) which they synthesized in 160 metaanalyses they impressively substantiate that there are way too many ‘positive’ results in the literature! Underlying reasons are reporting bias, including study publication bias, selective outcome reporting bias (where null results are omitted) and selective analysis bias (where data are analysed with different methods that favour ‘positive’ results). Study size was low (mean 16 animals), less than 1/3 of the studied randomized, or evaluated outcome in a blinded fashion, and only 39 of 4140 studies performed sample size calculations!
Call for international collaboration in preclinical research
Translational stroke medicine requires renewal, and international collaboration in preclinical research may be an important step to overcome hurdles impeding progress. The tremendous power of international research collaboration has been convincingly demonstrated in physics, and several transnational collaborations have already delivered proof of concept in the stroke field. The experience gleaned from such collaborations is paving the way for an exciting new era in stroke research, which strives to harness the multitude of benefits achievable through international collaboration. Now is the time for concrete action to advance the agenda and establish an international preclinical stroke network (click here for full article: Concerted Appeal for International Collaboration Stroke 2013)-
Overcoming negative (positive) publication bias
F1000 Research starts initiative to overcome ‘positive publication bias’ (aka ‘negative publication bias)’. Until end of August publication fees are waived for submission of Null results.
Only data that are available via publications—and, to a certain extent, via presentations at conferences—can contribute to progress in the life sciences. However, it has long been known that a strong publication bias exists, in particular against the publication of data that do not reproduce previously published material or that refute the investigators’ initial hypothesis. The latter type of contradictory evidence is commonly known as ‘negative data.’ This slightly derogatory term reflects the bias against studies in which investigators were unable to reject their null hypothesis (H0), a tool of frequentist statistics that states that there is no difference between experimental groups.
Researchers are well aware of this bias, as journals are usually not keen to publish the nonexistence of a phenomenon or treatment effect. They know that editors have little interest in publishing data that refute, or do not reproduce, previously published work—with the exception of spectacular cases that guarantee the attention of the scientific community, as well as garner extra citations (Ioannidis and Trikalinos, 2005). The authors of negative results are required to provide evidence for failure to reject the null hypothesis under numerous conditions (e.g., dosages, assays, outcome parameters, additional species or cell types), whereas a positive result would be considered worthwhile under any of these conditions . Indeed, there is a dilemma: one can never prove the absence of an effect, because, as Altman and Bland (1995) remind us, ‘absence of evidence is not evidence of absence’.
Several journals have already opened their pages to ‘negative’ results. For example, the Journal of Cerebral Blood Flow and Metabolism: Fighting publication bias: introducing the Negative Results section publishes such studies as a one-page summary (maximum 500 words, two figures) in the print edition of the journal, and the accompanying full paper online.
Power failure
In a highly cited paper in 2005, John Ioannidis answered the question ‘Why most published research findings are false’ (PLoS Med. 2, e124). The answer, in one sentence, is ‘because of low statistical power and bias’. A current analysis in Nature Reviews Neuroscience ‘Power failure: why small sample size undermines the reliability of neuroscience’ (advance online publication, Ioannidis is a coauthor) now focuses on the neurosciences, and provides empirical evidence that in a wide variety of neuroscience fields (including imaging and animal modeling) exceedingly low statistical power and hence very low positive predictive values are the norm. This explains low reproducibility (e.g. special issue in Exp. Neurol. with (lack of) reproduction in spinal cord injury research, Exp Neurol. 2012 Feb;233(2):597-605) and inflated effect sizes. Besides this meta-analysis on power in neuroscience research, the article also contains a highly readable primer on the concepts of power, positive predictive value, type I and II error, as well as effect size. Must read.
Diversity outcross mice
Most rodent models of disease (in stroke research, anyway) use young, healthy male, inbred mouse strains kept under specific pathogen free (SPF) conditions, restricted antigen exposure in their environment, and on a diet optimized for maximum reproduction rates (high in antioxidants, trace elements and other supplements, etc.). It is like studying cohorts of 12 year old male identical twins kept on an identical health diet in a single sealed room, without any contact to the outside world (the ‘plastic bubble’). What may be good for reproducible modeling, is potentially problematic for translational research, as patients often have comorbidities (e.g. hypertension and diabetes in stroke), already take various medicines, are elderly, and include females… Thus, external validity of the models often is low, at least partially explaining some of the failures when moving from promising new therapeutic strategies in rodents to real life patients in randomized clinical trials. Fortunately, external validity can be improved by studying comorbid animals at advanced age and of both genders. It is trickier in rodents to produce a mature immune system that had contact with pathogens and multiple antigens. The answer to reduced genetic diversity may be to use populations specifically developed to provide wide genetic variability, such as the diversity outbred population or the partially inbred collaborative cross strains developed by the Jackson Laboratory. However, in my field (stroke research), which is particularly hit hard by the ‘translational roadblock’ I have not seen a single study making use of these strains.
Nature Neuroscience initiative to improve reporting
Nature Neuroscience is currently undertaking an initiative to improve statistics and methods reporting.
Making methods clearer : Nature Neuroscience : Nature Publishing Group.
Following an initative of the NIH/NINDS (Landis et al.), Nature Neuroscience is testing a new scheme to improve reporting, and consequently to reduce bias and faulty statistics in work published in their journal. Authors have to fill out at very detailed checklist (stats checksheet nat neurosci source file ud) which is sent to the reviewers. Other journals had checklists before, but they were pro forma, and mostly ignored. This checklist looks very bureaucratic, and authors will hate it, but it contains all the relevant questions (stats including power; bias including blinding and randomization; ethics; detailed reporting of strains, animal husbandry, material; design of functional imaging studies, etc.). It forces authors to think about these issues, and if they haven’t done their homework before designing the experiments they either have to cancel submission, or fake entries into the sheet. Which would put them in clear violation of good scientific practice, compared to just not reporting that they used an underpowered design without blinding and randomization (which is the current practice)….
Nature Neuroscience should be commended for this initiative. Other journals will hopefully adopt this policy, and reviewers take the time to study the checksheets in order to request clarification from the authors or pull the plug on publication of flawed papers.
To infinity, and beyond! 