From peer review nuances to best practices
This paper analyzes the impacts of three specific nuances in peer review data—paper version, score version, and input format—on downstream tasks to establish best practices for both data providers and users.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
The Secret Life of Scientific Reviews
Imagine a world where scientists submit their big ideas to a panel of judges, much like a talent show. But instead of singing or dancing, they are presenting research papers. These judges, called "reviewers," read the work, write detailed feedback, and give it a score to decide if it should be published. This process is called peer review, and it's the gatekeeper of science. However, there's a catch: the paper a judge reads at the start of the process might look very different from the final version that gets published. Similarly, the score a judge gives before the author argues back might differ from the score they give after the argument.
Recently, researchers have started using super-smart computer programs, known as Large Language Models (LLMs), to help with this judging. These models can read a paper and write a review just like a human. But here's the problem: if we feed these computer programs the wrong version of a paper, or mix up the scores from different times, the results can be completely misleading. It's like asking a judge to rate a contestant's performance based on their audition tape, but then comparing that rating to the score they gave after the contestant fixed their mistakes on stage. To make sure these computer judges are actually learning the right things, we need to understand exactly which version of the paper and which score we are looking at.
The Paper's Big Discovery: Don't Mix Your Versions!
This paper is a wake-up call for anyone studying how computers handle scientific reviews. The authors, led by Lu Sheng, discovered that in the rush to use data for research, many people are accidentally mixing up different "versions" of the same story, leading to confusing and sometimes wrong conclusions. They looked at three main things that get mixed up: the paper version (the draft vs. the final polished copy), the score version (the score before the author argues back vs. after), and the input format (how the text is fed into the computer).
The Paper Version Puzzle
Think of a paper like a house under construction. The "initial draft" is the rough blueprint with some shaky walls. The "camera-ready version" is the finished house with fresh paint and a new roof. The authors found that papers that start off with lower scores tend to get the biggest makeovers. In fact, the "substance" parts of the paper (the methods and results) change the most, while the "framing" parts (like the introduction) change less.
Here is the tricky part: when they asked computer models to grade the rough draft and the finished house, the scores looked almost the same. It seemed like the computer didn't care about the improvements! But the authors dug deeper and found a secret ingredient: author information. When the authors' names and universities were included, the computer models gave higher scores to the finished house, likely because they were impressed by who wrote it (a "authority effect"). However, when they hid the authors' names, the computer models actually gave higher scores to the finished house because the content was genuinely better. The two effects canceled each other out, making it look like nothing changed. This means if you don't control for who wrote the paper, you might miss the fact that the paper actually got better.
The Score Version Trap
The authors also looked at what happens after authors get a chance to argue back (called a "rebuttal"). They found that nearly one-third (29.7%) of the reviews changed their overall score after this argument. Most of the time, the score moved by 0.5 points. This might sound small, but it's huge! About 65.6% of these changes happened right at the "decision line" (like moving from a 2.5 to a 3.0), which is the difference between a paper being accepted or rejected.
The danger here is mixing up the text and the score. Imagine a dataset where the computer reads the old review text (before the argument) but is asked to predict the new score (after the argument). The authors tested this and found that this mismatch makes the computer models look much better than they really are. In fact, the "best" computer model changed depending on whether you used the right scores or the mixed-up ones. If you use the wrong scores, you might think a model is a genius when it's actually just guessing based on a mismatched puzzle.
The Input Format Mystery
Finally, the team asked: does it matter if you feed the paper to the computer as plain text, a structured list (JSON), a formatted document (Markdown), or even as a picture of the page? For most of the computer models they tested, the format didn't change the score much. However, for one specific large model (gpt-oss-120b), the format made a big difference, with the structured JSON format leading to higher scores. This suggests that different computers "read" different formats in different ways, and we can't assume they all work the same.
What Should We Do?
The paper concludes with a set of "best practices" to stop this confusion. For the people who share data, they should save and label every single version of the paper and every score version, not just the final ones. For the people using the data, they need to double-check: "Am I using the draft or the final? Am I using the score before or after the argument? And what format is this in?"
By paying attention to these details, researchers can make sure their computer models are learning from the right information. The authors suggest that these "nuances" aren't just boring details; they are actually powerful tools to test how smart our computer models really are. If a model can tell the difference between a rough draft and a polished final, or if it can spot when a score has changed, it shows the model is truly understanding the content, not just guessing. So, the next time you see a study about AI reviewing papers, remember to ask: "Which version of the story are they telling?"
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.