Why Is Reviewer 2 Always the Villain? Myth, Bias, or Statistical Reality?
This study analyzes metadata from nearly 160,000 reviews across machine learning conferences and finds no statistical evidence supporting the "Reviewer 2" stereotype of inherent harshness or polarization, suggesting instead that the phenomenon is likely a measurement artifact or institution-dependent bias rather than a universal reality.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written by the author. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of academic science, the path from a new idea to a published paper is paved with a process called peer review. Before a study sees the light of day, it is sent to other experts in the field who read it carefully and offer their judgment. These experts, known as reviewers, act as gatekeepers, deciding whether the work is sound enough to be shared with the world. They assign scores, write detailed comments, and often recommend whether the paper should be accepted, rejected, or sent back for major changes. This system is designed to ensure quality, but it is also a source of great anxiety for authors. A single harsh comment can derail a career, and the process is often shrouded in secrecy, with the identities of the reviewers hidden from the author they are judging.
Over time, a specific cultural story has emerged within this system. Authors and scientists have developed a shared suspicion that the second reviewer listed on a report is uniquely difficult. The "Reviewer 2" has become a folk villain in the scientific community, a figure imagined to be disproportionately harsh, dismissive, or obstructive compared to their colleague. This stereotype suggests that if a paper has a second reviewer, that person is the one likely to kill the project. But is this a real pattern in how people behave, or is it just a feeling born of frustration? To answer this, a researcher needed to move beyond anecdotes and look at the actual numbers behind thousands of reviews to see if the second person on the list truly behaves differently than the first or the third.
A researcher set out to investigate this myth by examining a massive collection of real-world data from open science conferences. They gathered information from nearly 63,000 manuscripts and over 157,000 individual reviews. Their goal was to strip away the emotions and stories surrounding the process and look strictly at the scores and decisions. They focused on a specific question: do the people labeled as "Reviewer 2" actually give lower scores, use extreme ratings more often, or disagree more with their peers than the other reviewers evaluating the same paper? The researcher was careful to distinguish between the number assigned to a reviewer and the actual order in which they wrote their report, knowing that these two things are not always the same. It is important to note, however, that the data used in this study was concentrated predominantly in machine-learning conferences, specifically drawing 7,966 manuscripts from ICLR between 2018 and 2021. This means the findings are most applicable to these open-review settings and may not fully generalize to traditional, cross-disciplinary journals.
After analyzing the data with rigorous statistical methods, the study found no evidence to support the idea that Reviewer 2 is a special kind of villain. When the researcher compared the scores given by the second reviewer against the average scores of their peers, the difference was so small it was effectively zero. In fact, the data showed that the second reviewer was not systematically harsher, nor were they more likely to use the most extreme "reject" or "accept" ratings. The fear that the second person on the list is uniquely difficult appears to be a myth. The researcher also looked at whether the second reviewer's opinion carried more weight with the editors making the final decision. While a harsh review from anyone could lead to a rejection, there was no special "multiplier" effect for the second reviewer that made their opinion more powerful than the others.
The study did uncover some interesting details about how the process works, even if it debunked the main myth. The researcher found that the number a reviewer sees on their screen often does not match the order in which they actually submitted their work. In the dataset they examined, the public label of "Reviewer 2" matched the actual second person to submit a report in only about 30 percent of cases. This means that the "villain" authors complain about might not even be the second person to write a review; the numbering is often just a random assignment or a storage order that has nothing to do with the sequence of events. Furthermore, when the researcher looked at whether disagreements between reviewers were resolved after an author wrote a rebuttal, they saw that the group as a whole tended to move closer together in their opinions, but there was no sign that the second reviewer was the one changing their mind or causing the friction.
One of the most telling parts of the investigation involved testing whether knowing which reviewer gave which score could help predict the final outcome of a paper. The researcher built computer models to see if treating the second reviewer's score as a separate, special factor improved their ability to guess whether a paper would be accepted or rejected. The results showed that a simple average of all the scores worked just as well, and in some cases, slightly better, than a model that tried to give special weight to the second reviewer. This suggests that editors are likely looking at the collective wisdom of the group rather than fixating on a specific numbered position. The data indicates that the perception of a difficult second reviewer is likely a result of how humans remember negative experiences, rather than a statistical reality of the review system itself.
The study also looked at the dynamics of the review process over time. In a subset of papers where an author was allowed to respond to the reviewers, the researcher tracked how the scores changed. They found that the gap between different reviewers' opinions tended to shrink after the rebuttal stage, meaning the reviewers often found common ground. However, this convergence happened across the board and was not driven by a specific change in the behavior of the second reviewer. The researcher concluded that while the experience of dealing with a tough reviewer is real and painful for many authors, the idea that the second person on the list is uniquely to blame is not supported by the evidence. The system is complex, and disagreements happen, but they are not the result of a hidden bias against a specific numbered position.
Ultimately, this large-scale investigation serves as a reminder that our intuitions about how systems work can sometimes be misleading. The "Reviewer 2" phenomenon is a powerful story that resonates because it gives a face to the frustration of rejection, but the data shows that the face is not attached to a specific number. The harshness, the disagreement, and the high stakes of peer review are distributed across the entire group of experts, not concentrated in the second slot. By replacing the myth with measurement, the researcher has shown that the scientific community can better understand its own processes, focusing on the real sources of variation in judgment rather than chasing a ghost that does not exist in the numbers.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.