Triangulation! The Egyptians used it to build their pyramids. The Greeks developed a branch of mathematics out of it. Until the 19th century whole countries were charted in this way. Far into the 20th century ships have determined their position with it. To determine your position by triangulation you only need a set square and a protractor, which the surveyors call a theodolite, as well as the coordinates of two visible landmarks. It’s that simple!
Could it be that triangulation is also an important methodological approach in biology? A cure even for the replication crisis? Munafo and Smith recently postulated this in a commentary in Nature. Sociologists call it triangulation when they use two or more different methods to investigate one particular research question. If the results converge at one point, i.e. lead to the same result, this increases validity and credibility. Don’t we do this routinely in the experimental life sciences? Does the knock-out mouse have the same phenotype as one in which the signalling pathway was pharmacologically blocked? Do transcript and protein expression correlate with the phenotype?
Thus, basic biomedical research is familiar with ‘targeting’ a goal with different methods grounded in already established knowledge (the landmarks of the surveyor!). Are the results converging? Bingo, we have located the biological mechanism! Therefore it leaves many of us cold, if spoilsports with gradschool statistics argue that most studies in biomedicine must be false positive despite significant p-value. Because we don’t just rely on ONE result. Instead we triangulate by means of different approaches! In order to validate results, this might even be superior to replication. If something is simply repeated, it is not unlikely that a systematic error will be repeated too. This would make the result reproducible, but still not correct.
Were the skeptics wrong when calling out a crisis in biomedical research? Are we already doing the right thing? Continue reading
An article entitled “Growth in a Time of Debt” was published in 2010 by the highly respected Harvard economists Carmen Reinhart and Kenneth Rogoff. It dealt with the relationship between national economic growth and national debt. They reported on their discovery of an astonishing, globally observable correlation: As national debt rises, the economic growth of a nation initally also rises. If, however, the national debt exceeds 90 %, this ratio is reversed quite abruptly. Growth turns into contraction, and economic output then declines as debt rises further. The discovery of a “90 % debt threshold” hit like a bomb. Some suspect that the article was the basis for the European austerity policy after the 2008 financial crisis. What is certain, however, is that the paper was enthusiastically used by Western politicians to justify their restrictive fiscal policy. In 2013, Thomas Herndon, a student, reanalyzed the data of the Reinhart-Rogoff paper as part of a semester assignment. After some back and forth, the authors had given him the original Excel spreadsheet. And lo and behold, in a few minutes he found a number of serious errors in it! After correction, the debt threshold disappeared, and the data now appeared to prove the opposite, a steady, positive correlation between government debt and growth across the entire range! What do we learn from this? Apart from the fact that the fundamental error of Reinhart and Rogoff is of course the confusion of correlation with causation: Excel is not suitable for the analysis of complex scientific data. Even more importantly, scientists make mistakes, which can have serious consequences. Continue reading
Recently in a train station book shop I stood gaping in astonishment in front of a thematically highly specialized book display. It was the bowels-brain table. The books piled up on it promised enlightenment about how the bowel and in particular its contents influence us – yes – how, they verily steer our emotions. A selection of book titles: “Shit-Wise – How a Healthy Intestinal Flora Keeps us fit”; “Bowels heal brain heal body”; “Happiness begins in the bowels”, or “The second brain – How the bowels influence our mood, our decisions and our feeling of wellbeing”. Newspapers, magazines and the internet can also tell us this. The wrong bowel bacteria make us depressive – but the right ones make us happy … which is why yogurt helps against depression. Continue reading
U.S. economist Robin Hanson posed this question in the title of an article published in 1995. In it he suggested replacing the classic review process with a market-based alternative. Instead of peer review, bets could decide which projects will be supported or which scientific questions prioritized. In these so-called “prediction” markets, individuals stake “bets” on a particular result or outcome. The more people trade on the marketplace, the more precise will be the prediction of outcome, based as it is on the aggregate information of the participants. The prediction market thus serves the intellectual swarms. We know that from sport bets and election prognoses. But in science? Sounds totally crazy, but it isn’t. Just now it is making its entry into various branches of science. How does it function, and what does it have going for it? Continue reading
With a half-page article written about him and his study, an Israeli radiologist unknown until then made it into the New York Times (NYT 2009). Dr. Yehonatan Turner presented computer-tomographic scans (CTs) to radiologists and asked them to make a diagnosis. The catch: Along with the CT a current portrait photograph of the patient was presented to the physicians. Remember, radiologists very often do not see their patients, they make their diagnosis in a dark room staring at a screen. Dr. Turner in his study used a smart cross-over design: He first showed the CT together with a portrait photograph of the patient to one group of radiologists. Three months later the same group had to make a diagnosis using the same CT, but without the photo. Another group of radiologists were first given only the CT and then, three months later the CT with photo. A further control group examined only the CTs, as in routine practice. The hypothesis: When a radiologist is exposed to the individual patient, and not only to an anatomical finding on a scan, she will be more conscious of her own responsibility, hence findings will be more thorough and diagnosis more accurate. And in fact, this is what he found. The radiologists reported that they had more empathy with the patient, and that they “felt like doctors”. And they spotted more irregularities and pathological findings when they had the CT and photo in front of them than when they were only looking at the CT (Turner and Hadas-Halpern 2008).
So how about showing researchers in basic and preclinical biomedicine photos of patients with the disease they are currently investigating in a model of the disease? Continue reading
It struck at the end of July. A ‘scandal’ in science shook the Republic. Research by the NDR (Norddeutscher Rundfunk), NDR (Westdeutscher Rundfunk) and the Süddeutsche Zeitung revealed that German scientists are involved in a “worldwide scandal”. More that 5000 scientists in German universities, institutes and federal authorities had, with public funds, published their work in on-line pseudoscientific publishing houses that do not comply with the basic rules and for assuring scientific quality. The public and not just a few scientists heard for the first time about “predatory publishing houses” and “predatory journals”.
Predatory publishing houses, whose presentation in phishing mails is quite professional, offer scientists Open Access (OA) publication of their scientific studies at a cost, whereby they imply that their papers will be peer reviewed. No peer review is carried out, and the articles are published on the web site of these “publishing houses”, which however are not listed in the usual search engines such as PubMed. Every scientist in Germany finds several such invitations per day in his or her e-mails. If you are a scientist and receive none, you should be worried about it. Continue reading
Slides of my talk at the FENS Satellite-Event ‘How reproducible are your data?’ at Freie Universität, 6. July 2018, Berlin
- Let’s get this out of the way: Reproducibility is a cornerstone of science: Bacon, Boyle, Popper, Rheinberger
- A ‘lexicon’ of reproducibility: Goodman et al.
- What do we mean by ‘reproducible’? Open Science collaboration, Psychology replication
- Reproducible – non reproducible – A false dichotomy: Sizeless science, almost as bad as ‘significant vs non-significant’
- The emptiness of failed replication? How informative is non-replication?
- Hidden moderators – Contextual sensitivity – Tacit knowledge
- “Standardization fallacy”: Low external validity, poor reproducibility
- The stigma of nonreplication (‘incompetence’)- The stigma of the replicator (‘boring science’).
- How likely is strict replication?
- Non-reproducibility must occur at the scientific frontier: Low base rate (prior probability), low hanging fruit already picked: Many false positives – non-reproducibility
- Confirmation – weeding out the false positives of exploration
- Reward the replicators and the replicated – fund replications. Do not stigmatize non-replication, or the replicators.
- Resolving the tension: The Siamese Twins of discovery & replication
- Conclusion: No scientific progress without nonreproducibility: Essential non-reproducibility vs . detrimental non-reproducibility
- Further reading
Open Science Collaboration, Psychology Replication, Science. 2015 ;349(6251):aac4716
Goodman et al. Sci Transl Med. 2016;8:341ps12.