On March 17th, just as many countries were taking draconian measures to contain the SARS-COV-2 pandemic, the Greek-American meta-researcher and epidemiologist John Ioannidis, whom I often quote in my posts proclaimed a “fiasco in the making‘! With strong language and a few ad hoc estimations of COVID fatality rates he warned that based on poor data or no evidence at all politicians might inflict incalculable damage on society, possibly much worse than what a virus, putatatively as dangerous as influenza, could cause. As one of the most highly cited researchers in the world and a vocal critic of quality problems in biomedicine, his COVID related interviews, opinion pieces and articles since then have received a great deal of attention, in the scientific community, in the lay press, and especially among his worldwide fan base.

Shortly after his commentary on STAT, Ioannidis followed up with data from two scientific studies. Based on the evaluation of official mortality data from various countries, he concluded that for most people the probability of dying from COVID is about as low as having a fatal accident on the way to  work. Together with colleagues from Stanford University in a serological study he then found that in Santa Clara County, California, more than 50 times more people were infected by the virus than officially confirmed by PCR. The implication: An infection fatality rate much lower than calculated based on official data..

No wonder that he quickly became a key scientific witness for a relaxation or even lifting of the lockdown. The results of both studies were eagerly taken up by the conservative media in the US. He became a sought-after interview partner, especially in right-wing media such as FOX News. At the same time, however, both studies (published as preprints) were heavily criticized by infectiologists, epidemiologists, and statisticians.

Those who, like myself, consider John Ioannidis the undisputed guardian of scientific rigor, were perplexed. Not so much because of the appropriation of his results and conclusions by reactionary media – correct arguments don’t become flawed by being voiced to obscurants or quoted by them. Nor is it because Ioannidis with his statements opposed the scientific and political mainstream, this has always been his trademark. No, the shitstorm  escalating into a ‘Ioannidis affair’ was ignited by obvious methodological weaknesses, which, in sum, called into question Ioannidis’ arguments for a misjudgment of the dangers of SARS-COV-2. He was accused of having authored studies and intervened prominently in the public debate with results which quite clearly did not meet his own quality standards. It was gloatingly pointed out that he himself now had provided the ultimate proof for the validity of his most famous article published in 2005, entitled ‘Why most published research findings are false‘!

But is John Ioannidis’ main argument, despite methodological weaknesses, perhaps still correct after all? At the heart of this dispute is the question how dangerous SARS-COV-2 really is, and whether the draconian measures against the virus could end up being more harmful than the virus itself. Many point to New York or Northern Italy, or refrigerated containers full of corpses for a simple answer.

But it is not as simple as that, because science did not yet (and still doesn’t) have clear answers to the exact degree of SARS-COV-2 infection prevalence, to the ultimate causes of COVID related deaths, and above all to the impact of collateral damage. What is clear is that not only the direct morbidity and mortality of the virus must be taken into account. The overloading of (unprepared or already dysfunctional) health care systems has to be considered as well, including the effects of avoiding necessary medical treatments because of the fear of getting infected in the hospital. In addition, to the psychological effects of the lockdown and the late effects of school closures. And then of course the effects of the economic crisis or even a possible collapse of the economy in the wake of a lockdown. Now unemployment and poverty have already massively increased in many countries, with their well known consequences for health and life expectancy.

But how solid was the evidence of Ioannidis’ studies, which were celebrated by FOX News and torn apart by many experts? In one, he calculated the relative and absolute risk of dying from COVID-19 at the population level. Critiques scorned that at the beginning of an epidemic, when the prevalence of a disease is by definition low, it makes little sense to calculate the absolute (or even relative) risk of dying from the disease. What is much more important in the early phase of a pandemic is not the absolute mortality risk of infection, but the capacity of the health care system, and the extent of collateral damage of non-pharmacological interventions (e.g. social distancing, lockdown, etc.). Ioannidis’ focus on COVID mortality has been compared by Mark Lipsitch (Harvard), one of the world’s leading infection epidemiologists, to “calculating the absolute risk of mortality in the first 3 days after a cancer diagnosis“.

The second study, already mentioned above, found substantially more viral infections in the Californian study area than official statistics. This is because, as everywhere else, only those who had typical disease symptoms or contact with infected persons were tested by PCR. This necessarily means that we are missing most of those who do not become seriously ill despite being infected, and there seems to be a lot of them with SARS-COV-2. This naturally reduces the calculated infectious mortality, which is calculated by dividing the number of people who die from the virus by the number of people infected with it. The preprint was also torn up immediately after publication. Criticism focused at selection bias, statistical evaluation, and (inadequate) validation of the test kit. If the specificity of a test is not 100 %, and the feature under investigation is rare (‘low prevalence’), a high false positive rate must be assumed. The Santa Clara study found a seroprevalence of around 3%. However, due to imperfect test specificity and low prevalence, it cannot be excluded that most positives could have been false. To add insult to injury, it turned out that David Neeleman, the founder of the airline JetBlue was one of the sponsors of the study which was not disclosed. The owner of an airline naturally has a massive interest in the relaxation of travel restrictions.

The far reaching conclusions that “These new data should allow for better modeling of this pandemic and its progression under various scenarios of non-pharmaceutical interventions” and “Population prevalence estimates can now be used to calibrate epidemic and mortality projections” can hardly be justified given the limitations and methodological weaknesses of both studies.

In the meantime, however, revised preprints of both studies have been published, with additional data and analyses added. The pandemic has also progressed further, and the mortality rates, which had been updated, have remained essentially unchanged to initial version of the preprint. Ioannidis considered all criticism to be sufficiently addressed. As up to now, a number of other studies have found mortality and seroprevalence rates in a similar range.

Although the evidence base for estimating mortality, prevalence and infectiousness of SARS-COV-2, as criticized by Ioannidis already in March, has meanwhile matured, epistemic uncertainty is still very high. At present the greatest uncertainty concerns collateral damage of the containment measures. First studies, mostly published as preprints, indicate that these will be massive, e.g. in the fields of cardiovascular diseases, cancer, and mental disorders. We now see a high excess mortality rate, interestingly except in Germany, which may be highly informative. What is particularly worrying, however, is that the majority of fatalities are in elderly people, especially in nursing homes. Notably, the excess mortality is partly due  to a considerable proportion of ‘other causes’, i.e. not directly related to the virus. One can die from COVID19 without being infected! Thus, the focus should no longer be on the ‘natural phenomenon’ of a viral pandemic, but rather on dysfunctional health care systems, poverty, health care shortages, conditions in nursing homes, etc. The significant differences in COVID19 mortality between cities, regions, and countries are not so much biological as social and political.

So what is my verdict? John Ioannidis has never missed an opportunity to emphasize that politicians had acted correctly when enacting immediate draconian defensive measures in the face of an acute, unclear, and potentially disastrous situation. He was indeed advocating a “better safe than sorry approach in the absence of good data”. The results of his studies are now no longer as controversial: seroprevalence values in the range of the Santa Clara County study were reported from other study regions. The low mortality of SARS-COV-2 among those under 65 years of age without concomitant diseases in regions with functioning healthcare systems is now common knowledge. We do not yet know much about the secondary damage caused by the containment measures, but there are signs that they will be catastrophic, as Ioannidis predicted as early as March.

Nevertheless, a shadow falls on the (demi)god of scientific reform. Despite a weak evidence base which he himself rightfully criticized, he in the same breath with drastic words suggested that we are dramatically overreacting. Supported by methodologically problematic and presumably biased own studies, he then immediately put himself in the limelight before – or rather during – a scientific discussion of his studies took place. He must have been aware that his appearances would be politically instrumentalized. The gods of the Greeks were immortal, but had the form of human beings, with their negative qualities and weaknesses. Ioannidis has shown us that even the most distinguished critics of scientific rigor can make mistakes.

Interestingly, and even closer to home, a very similar story is currently unfolding concerning another  scientific ‘(demi)god’, the head of the virology department of the Charite, Christian Drosten. Undoubtedly, he has an impeccable record as corona researcher, open science advocate, and public communicator of complex biology. Now his recent publication (on an institutional server) ‘An analysis of SARS-CoV-2 viral load by patient age’ has come under severe attack. The criticism, and calls for retraction, sound familiar: Questionable statistics, overinterpretation of data, and disappointingly little willingness to admit own mistakes.

Besides insights into the psychology of high profile researchers, and an overheated research enterprise under maximum public and political pressure and observation, what do we learn from all this? On the one hand, we see the strengths and weaknesses of preprints at work. Their strength lies in the fact that immediately after publication an intense real-time discussion of  scientific studies can take place. Which then enables corrections and improvements to be made. As a revision of the preprint, the work is then again subject to the criticism of a large number of experts from diverse fields of expertise. An improved article is then submitted for submission and regular peer review in a specialist journal. This is currently the case in about 70% of preprints. Perhaps such intensively discussed manuscripts then no longer need to find their way into a regular journal. The natural sciences, above all mathematics and physics, have been using  preprints  (‘arXiv’) since the early 1990s. As a result, the majority of preprints in these sciences are no longer submitted to journals. It would be a waste of time.

The weakness of the preprints is of course that they are not reviewed by peers. They might contain complete nonsense, or be brilliant: only experts can judge this, and even they sometimes find it difficult. Thus, preprints potentially provide the material for obscurants, extremists, or politicians with dangerous agendas. But let’s be honest: regular peer review is not a guarantee for quality and accuracy of results. Remember Wakefield’s Lancet study on vaccination and autism, or the Science and Nature papers of Schön, Stapel, Obokata?.

But maybe this is not even a ‘weakness’ of preprints! On the contrary: Preprints are currently giving us and the general public a practical demonstration that science does not provide definitive truths in the form of publications. That science is difficult, that it is organized skepticism, that it makes mistakes, that it is always on the move, that it questions its own results at any time and revises them as soon as mistakes are discovered or better evidence is available. This is now demonstrated by preprints, but it also applies to peer-reviewed articles and even textbooks.

Moreover, the Ioannidis ‘affair’ teaches us that science in times of universal crisis must not invoke ‘research exceptionalism’, as Alex London and Jonathan Kimmelman recently formulated in an article in Science. Scientific and ethical standards must not be lowered under time pressure, as happened in Ioannidis’ and Drosten’s recent studies – on the contrary, they must be raised.  Bad data is not better than no data!

 

Disclosure: John Ioannidis is currently Einstein BIH Visiting Fellow at the Berlin Institute of Health.

An earlier German version of this post has been published as part of my monthly column in the Laborjournal: ‘Der Fall Ioannidis’

 

Update: On July 24 2020 BuzzFeed ran this story by Stephanie M. Lee :

An Elite Group Of Scientists Tried To Warn Trump Against Lockdowns In March
John Ioannidis’s controversial studies claim that the coronavirus isn’t that big a threat. Before the Stanford scientist did any of them, he wanted to take that message to the White House.