We all have been there: After a long wait and increasing tension, an email arrives from the editotial office. With an increasing heart rate we open the email. “Thank you for submitting your manuscript to our Journal. Your manuscript was sent for external peer review, which is now complete. Based on our evaluation and the comments of external reviewers, your manuscript did not achieve a sufficient priority for further consideration, and we have decided not to pursue your manuscript for publication. While we understand you may be disappointed with this decision, we hope the reviewer comments will help you revise your manuscript and submit it to another journal. Thank you for the privilege of reviewing your work.’ After the initial shock, we take a look at the reviews in the attachment. Reviewer 1 found the work quite good, some minor issues, all fixable, with well-meaning suggestions. But Reviewer 2! Has he even read the article? Did he confuse it with another paper on his desk? In any case, this unknown ‘expert’ was totally incompetent, but still dared to pour several pages of slurry over 3 years of our hard work and its highly relevant results.

On the other hand, we all have also experienced this: tough but constructive criticism from the reviewers, who identified one or the other problem that we ourselves had overlooked or did not want to admit. With excellent advice on how to substantially improve our study and article after an additional experiment and a revision of the manuscript!

We all have probably had mixed experiences with peer review, and are aware of the many weaknesses of peer review. But we must accept it because if you want to make a living in science you have to live with it. We concentrate on our experiments and paper writing, rather than reflecting  on possible alternatives or supplements to scientific quality control. Would you choose the option ‘Send out for review’ if you could also choose ‘Publish immediately’ when submitting your paper?

Interestingly, preprints, which are taking biomedicine by storm (after having been a standard mode of publication in mathematics and physics) offer exactly this option. Corona and the associated public education campaign on ‘How does science actually work’ has made publishing without review popular not only to COVID researchers, but to all scientists. Even the general public by now knows what a ‘preprint is’, and peer review for that matter. All this has fueled, for the first time, a general discussion about the merits, or perils of peer review and its alternatives.

Just to get it out of the way, much has been said and written about the weaknesses of peer reviewed: When manuscripts are submitted, the horse is already out of the barn – the study has already been conducted, substantial suggestions for improvement must come too late. Peer review delays the dissemination of new findings. It promotes mainstream research, because reviewers tend to favour the cozyness of their own horizon. Peer review in most cases is completely intransparent, and thus promotes cronyism and vendettas. It allows the theft of ideas. Its results are not reproducible, its quality erratic. No matter how bad a work is, after multiple submissions to an increasing number of journals with declining reputation (= impact factor), it will always be published. It does not prevent scientific fraud. It eats up immense resources – for us as researchers due to multiple revisions and submissions, for us as reviewers because it takes quite some time to write a good review, and for us as taxpayers. Because we finance the subscription fees of libraries or Open Access fees (APC). These fees are needed to fuel the machinery that keeps the review process at the journals going. The list of problems of peer review don’t end here. And there is solid, scientific evidence for all this.

A considerable effort when writing a paper is spent to influence or even manipulate the peer review process. Papers are cited and evidence is selected that flatters potential reviewers. We endlessly debate which reviewers should be suggested or better excluded. Journals are selected according to whether we have friends on the editorial board, etc. Those who master this process, usually the advanced and senior researchers, have a clear advantage in this game. But early career researchers are trying to keep up: Meet the editor’ sessions are one of the best attended events at conventions. They hope to get tips and tricks to improve their chances for acceptance in the respective journal.

Those who consider all this to be normal are already fully socialized in the scientific enterprise. All others, however, should ask themselves what these activities have to do with science. And why, despite all its problems, peer review in its present form (still) exists at all. Rumor has it that it is as old as modern science itself. That it is virtually embedded in the DNA of the scientific method. But this is not true. The peer review of the gentleman scientists of the 17th century, of the Boyle’s and Hooke’s, had very little to do with todays procedures. And do you think that ‘The Molecular Structure of Nucleic Acids’ by Watson and Crick was externally reviewed before it was published in Nature in 1953? Of course not, because peer review did not exist at that time. Peer review in its present form evolved only 20 or 30 years later. And this at a time when much less was published and the methodology used was much less complex than today. ‘The system’ was also less competitive than today. The ‘luminaries’ of the field often published little, and if they did, it was in the journals of their professional societies. In a given research field, everyone knew each other. Excellence was not measured by the number of Nature papers. Scientific feuds were fought with an open visor, and often also with harsh language.

Peer review has become the most important quality control instrument in science. When spectacular cases of fraud are reported in the lay press we argue that this only confirms the rule, i.e. that it was an exception. Isn’t it strange that we claim that peer review is key to the scientific process, but make no effort to professionalize it? Although there is a lot to consider and even more that can go wrong, there is very little training in it, if at all. We start reviewing as soon as the first request from a journal comes in.

In reality, the most essential task of peer review today is no longer quality control, but the operationalization and quasi-objectification of the hierarchy of scientific journals. This is done by maintaining the selectivity necessary for this hierarchy. The proliferation and industrialization of the production of knowledge and its dissemination in the form of scientific articles, as well as the immense increase in the complexity of topics and methods has led to an overwhelming number of authors, reviewers and editors. The era of renaissance men, or ‘generalists’ is over. Reviewers can now only assess select aspects of the work presented to them. In order to check the quality of the data, if it and the analysis scripts used are made available to them at all, reviewers would have to struggle with a manuscript for days. Just look at his paper in Cell metabolism, which I recently read  a Cell Metabolism paper where data was scattered across 15 images in the supplement alone, some of which had panels the size of 26 stamps, labeled a-z! Does anyone seriously believe that this information was, or even could be, scrutinized by a referee?

The idea of peer review as a thorough discourse among scientists about research results is noble and plausible, and has certainly proven itself for a few decades. For a good reason, peer review has become a standard and deserves in principle its nimbus of a ‘quality control instrument’. But today it is overstrained. Not the least due to the hypercompetition that now prevails in the scientific community and the quantifiability of the prestige of journals through the impact factor. This has led to the commercialization of the product ‘scientific article’. Scientific articles have become the most important currency in the competition among scientists. Publishers make a living from it by setting the exchange rate of this currency in their competition with each other. It no longer depends so much on what is written in an article, but rather where it was published. The prestige of the journal ennobles its content and at the same time vouches for its quality independently of its content. In this process, the peer review has the function of a quasi-objective selection instrument.

That papers are improved or that scientific garbage is sorted out has become a side issue. Of course, if an article finds the right reviewers, we might receive relevant and constructive advice. Which protects zs from making a fool of ourselves by our own supposedly sensational results and prevents us form publishing flawed results or exaggerated conclusions. Peer review sometimes involves scientific discourse at its best!

So would it be possible to make use of the strengths of peer review but avoid its weaknesses? I think so! A simple but highly effective approach is to review studies before they actually begin, making them ‘Registered Reports’. (further details see here). Transparency can be added in all forms of review by disclosing the correspondence between the journal and the reviewers. Transparency can be further increased by ‘Open Review’, i.e. reviews without the protection of anonymity. But does’nt that lead to courtesy reviews and old boys networks? I don’t think so, because everyone would be able to see it! Incompetent or unfair comments become less frequent, because they would backfire to the reviewer as their name is attached to it. A potential disadvantage of open reviews, however, might be that young scientists in particular would have to fear that critical comments could endanger their careers. Which tells us something about the system. But there could be a remedy for this, e.g. by co-reviewing with more established scientists. Their names would then be published, of course with reference to the collaboration, and the young scientists could still get credit for their work through platforms such as Publons (https://publons.com).

But in my view the most exciting and often even best reviews are currently published in social media, for both preprints and regular articles. True quality control has been outsourced to science Twitter: a almost all manipulated or otherwise fraudulent articles were exposed by skeptical readers and then brought into international discourse via Twitter, PubPeer or blogs. Many of the COVID preprints were reviewed by the ‘swarm’. Indeed, often authors consider the most relevant comments in revisions of their preprints or in the submission to a peer review journal.

Some journals already successfully practice some or even all of the above mentioned modifications of peer review. These include Elife, F1000Res, EMBOJ, PLOS Journals, BMJ and PeerJ. However: In order for all of this to have a real impact and for peer review to once again become an instrument of critical and constructive exchange between scientists, a few other things have to change at the same time. First and foremost: We need to write fewer, but better articles. Their content and quality should become more important criteria in the evaluation of scientists than the names of the journals in which they publish. Long live peer review!

 

A German version of this post has been published as part of my monthly column in the Laborjournal: http://www.laborjournal-archiv.de/epaper/LJ_20_10/30/index.html

 

Just a few references:

Chambers, C. D., & Tzavella, L. (2020, February 10). Registered Reports: Past, Present and Future https://osf.io/preprints/metaarxiv/43298/

Dirnagl U (2020) Preregistration of exploratory research: Learning from the golden age of discovery. PLoS Biol 18(3): e3000690. https://doi.org/10.1371/journal.pbio.3000690

Editorial. Nature will publish peer review reports as a trial. Nature 578, 8 (2020) https://www.nature.com/articles/d41586-020-00309-9

Elsevier/Sense about Science. Quality, trust & peer review: researchers’ perspectives 10 years on. Sept 2019 https://senseaboutscience.org/wp-content/uploads/2019/09/Quality-trust-peer-review.pdf

Glonti, K., Cauchi, D., Cobo, E. et al. A scoping review on the roles and tasks of peer reviewers in the manuscript review process in biomedical journals. BMC Med 17, 118 (2019). https://doi.org/10.1186/s12916-019-1347-0

Heesen R, Bright LK. Is Peer Review a Good Idea?, The British Journal for the Philosophy of Science, axz029, https://academic.oup.com/bjps/advance-article/doi/10.1093/bjps/axz029/5526887

Jump P. Slay peer review ‘sacred cow’, says former BMJ chief. Times Higher Education 21.4.2020 https://www.timeshighereducation.com/news/slay-peer-review-sacred-cow-says-former-bmj-chief/2019812.article#

Mazer B. To Maintain Trust in Science, Lose the Peer Review. Medscape 19.2.2019 https://www.medscape.com/viewarticle/908647?src=WNL_infoc_190223_MSCPEDIT_Ward&uac=124874SX&impID=1892056&faf=1

Peterson GI. Postpublication peer review: A crucial tool. Science  16 Mar 2018:Vol. 359, Issue 6381, pp. 1225-1226 https://science.sciencemag.org/content/359/6381/1225.2

Smith RJ. Peer review: a flawed process at the heart of science and journal. J R Soc Med. 2006 Apr; 99(4): 178–182. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1420798/#!po=79.1667

Vazire S. Peer-Reviewed Scientific Journals Don’t Really Do Their Job. WIRED 26.6.2020 https://www.wired.com/story/peer-reviewed-scientific-journals-dont-really-do-their-job/