💬 NLP

Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time

Challenging the popular narrative of declining standards, this paper introduces a new framework for quantifying review quality and demonstrates through a cross-temporal analysis of major AI and linguistics conferences that median review quality has remained stable over time.

Original authors: Ilia Kuznetsov, Rohan Nayak, Alla Rozovskaya, Iryna Gurevych

Published 2026-01-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ilia Kuznetsov, Rohan Nayak, Alla Rozovskaya, Iryna Gurevych

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: Are Reviews Getting Worse?

Imagine you are running a massive, global talent show. Every year, thousands of singers (researchers) submit their songs (papers) to be judged by a panel of experts (reviewers). Recently, there has been a lot of gossip in the audience: "The judges are getting lazy! They aren't listening closely anymore. The quality of the feedback is dropping because there are too many singers and not enough judges."

This paper asks a simple question: Is this gossip true? Do the reviews actually get worse over time, or is that just a feeling?

The Problem: You Can't Compare Apples to Oranges

To answer this, the researchers tried to measure "review quality." But they hit a snag. Imagine trying to compare the quality of a sandwich from 2018 to a sandwich from 2025.

In 2018, the sandwich might have been served on a napkin with no instructions.
In 2025, it comes in a fancy box with a checklist of ingredients.
Some judges write long paragraphs; others write bullet points.
Some judges write in a specific format; others write freely.

Because the "forms" and styles changed so much over the years and across different conferences (like ICLR, NeurIPS, and ACL), it was impossible to fairly compare them. It was like trying to measure the height of people using a ruler for one group and a tape measure for another.

The Solution: The "Review Translator"

To fix this, the authors built a universal translator for reviews.

Flattening: They took all the different forms and flattened them into one long, continuous block of text.
Itemization: They used a smart AI (a Large Language Model) to act like a professional editor. This AI took the messy text and reorganized it into standard sections: Summary, Strengths, Weaknesses, and Other.

Think of this as taking every sandwich, regardless of how it was served, and slicing it into standard pieces so you can taste the bread, the meat, and the cheese separately. This allowed them to compare reviews from 2018 directly with reviews from 2025.

How They Measured "Quality"

Once the reviews were standardized, the team measured them using three main "taste tests":

Substantiveness (Is there enough meat on the bone?):
- Lightweight: How long is the review? (Counting characters).
- Deep: How many distinct points or "items" did the reviewer make?
- Analogy: A short review is like a tiny cracker; a long one is a full meal. They counted how many "bites" of information were in the review.
Actionability (Can the singer actually fix their song?):
- Did the reviewer give specific instructions? (e.g., "Change Figure 3" vs. "This is bad").
- Analogy: A good review is like a coach saying, "Your left foot is too far forward." A bad review is just shouting, "You're off!" The authors measured how many specific "fix-it" requests were in the text.
Grounding (Did they look at the actual paper?):
- Did the reviewer point to specific parts of the paper? (e.g., "See page 4, line 10").
- Analogy: A grounded review is like a detective pointing to evidence on the table. An ungrounded review is just guessing without looking at the clues.

The Results: The Gossip Was Wrong

After analyzing thousands of reviews from top computer science conferences over several years, the data told a surprising story:

The median quality of reviews did NOT decline.

Despite the fear that reviewers are burning out or that AI is making them lazy, the "average" review today is just as helpful, detailed, and grounded as the reviews from a few years ago. The "gossip" of decline wasn't supported by the numbers.

Why Does It Feel Like Quality is Dropping?

If the quality is the same, why does everyone feel like it's getting worse? The authors offer a few theories (hypotheses):

The "More People, More Noise" Theory: Even if the average quality stays the same, as the number of submissions grows, the total number of bad reviews increases. If you have 100 reviews and 5 are bad, that's 5%. If you have 10,000 reviews and 500 are bad, that's still 5%, but suddenly 500 people are angry. The volume of bad experiences makes it feel like everything is broken.
The "Worst Got Worse" Theory: Maybe the average is fine, but the absolute worst reviews are becoming even worse, and those are the ones people talk about on social media.
The "We Just Don't Know" Theory: Maybe the quality did drop in ways the authors couldn't measure (like the "soul" of the review), but their current tools couldn't see it.

The Takeaway

The paper concludes that we shouldn't panic and assume the system is collapsing. The "average" review is holding its ground. However, because the number of papers is exploding, we need to work harder to ensure that everyone gets a good review, not just the average one.

The authors also built a toolkit (code and data) that other researchers can use to keep checking this "health" of the review system in the future, ensuring we have facts, not just feelings, to guide us.

Technical Summary: Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time

Problem Statement

Peer review is the cornerstone of modern scientific validation, yet a prevailing narrative suggests a decline in review quality driven by rising submission volumes, reviewer scarcity, and the irresponsible use of Large Language Models (LLMs). Despite widespread concern, empirical evidence regarding the trajectory of review quality is largely anecdotal. The primary challenges in studying this trend are:

Measurement Difficulty: Review quality is difficult to formalize, and human judgment is prone to bias and does not scale.
Data Heterogeneity: Review formats, guidelines, and structures vary significantly across venues (e.g., ICLR, NeurIPS, ACL) and over time, making consistent cross-temporal comparison impossible with existing methods.
Lack of Framework: There is no holistic, computational framework for the multi-dimensional analysis of review quality at scale.

Methodology

The authors propose a new framework for the evidence-based comparative study of review quality, applied to major AI and machine learning conferences (ICLR, NeurIPS, and *ACL/ARR). The methodology consists of three core components:

1. Review Unification

To address structural diversity, the authors introduce a two-step unification process:

Flattening: All text fields and section headers from a review form are concatenated into a continuous text string ( $r_f$ ), discarding non-textual fields.
Itemization: A novel LLM-based approach (using GPT-5.2 in a zero-shot setting) transforms the flattened text into a set of self-contained bullet-point items ( $r_i$ ). These items are assigned to predefined sections: Summary, Strengths, Weaknesses+ (aggregating weaknesses, questions, and suggestions), and Other.
Validation: The itemization process was evaluated for content preservation (hallucination and omission) and section assignment accuracy, achieving high precision (0.95) and recall (0.88) in sentence-level alignment, with human raters confirming no hallucinations or omissions.

2. Multi-Dimensional Quality Schema

The authors define review quality based on utility for editors and authors, operationalized through three criteria and six specific measurements (Table 1):

Substantiveness:
- ITX (Item Count): The count of core informational items in Summary, Strengths, and Weaknesses+ sections.
- LEN (Length): Character count of the flattened review (lightweight proxy).
Actionability:
- ACT (Actionability Score): An LLM-based score (1–5) assessing how easily suggestions can be translated into specific actions.
- REQ (Request Count): A lightweight count of explicit request sentences (e.g., "Please add...") identified via a pragmatic tagging classifier.
Grounding:
- GND (Grounding Score): An LLM-based score (1–5) measuring the extent to which comments are anchored to specific parts of the paper.
- EXL (Explicit Links): A lightweight count of explicit links from the review to the paper (e.g., "Fig. 1", "Sec. 2").

3. Aggregate Scoring and Analysis

Normalization: All metrics are normalized to [0, 1].
Composite Score ( $Q$ ): An aggregate review quality score is calculated as the non-weighted average of all normalized measurements.
Data Scope: The study analyzes a random sample of 1,000 reviews per campaign from ICLR (2018–2025), NeurIPS (2021–2025), and ARR (2022–2025), excluding desk-rejected and withdrawn papers.

Key Contributions

Holistic Framework: The first computational framework for the large-scale, multi-dimensional study of review quality across venues and time.
Review Unification: A novel, meaning-preserving approach to standardizing diverse review structures using LLMs, enabling cross-temporal analysis.
Quality Schema: A multi-dimensional schema combining lightweight (efficient) and LLM-based (deep) measurements for Substantiveness, Actionability, and Grounding.
Empirical Analysis: The first large-scale study applying this schema to analyze the evolution of review quality in top-tier CS conferences.

Results

Correlation and Measurement Types

All measurements are positively correlated, indicating non-contradictory quality dimensions, yet they are not perfectly correlated, suggesting they are complementary.
Lightweight measurements (LEN, REQ, EXL) show moderately strong correlation ( $\rho \approx 0.6$ ) with LLM-based measurements (ACT, GND) in aggregate, suggesting lightweight proxies can approximate resource-intensive metrics at scale.

Temporal Trends

No Consistent Decline: Contradicting the popular narrative, the analysis reveals no consistent decline in median review quality across venues and years.
Fluctuations: While some fluctuations exist (e.g., a notable increase in median $Q$ for ICLR-2021, ICLR-2025, and NeurIPS-2025), bootstrap tests indicate many year-to-year differences are not statistically significant or practically negligible.
Accepted vs. Rejected: In ICLR, where data is fully open, the median quality for accepted papers (0.24) is only marginally higher than for rejected papers (0.23).

Distribution

The distribution of the aggregate score $Q$ is skewed due to positive outliers in count-based metrics, with a small subset of extremely low-scoring reviews (e.g., placeholders or refusals).

Significance and Claims

The paper claims that the "decline in review quality" narrative is not supported by empirical data when analyzed through a systematic, multi-dimensional framework. The authors posit that the perception of decline may stem from:

Statistical Artifacts (H5): As the total number of reviews grows, the absolute number of low-quality reviews increases proportionally, leading to more authors experiencing poor reviews and amplifying negative sentiment on social media.
Incomplete Measurement (H3): Current metrics may miss specific dimensions of quality, or historical data (pre-2021) might show different trends that are currently inaccessible.
Worst-Case Scenarios (H4): While the median may be stable, the quality of the worst reviews might have degraded, though this requires further investigation with new metrics.

The authors emphasize that their framework is exploratory and intended to facilitate future empirical studies rather than serve as a direct tool for policy design or automated review detection. They call for the community to move beyond anecdotal evidence and utilize scalable, standardized metrics to understand and improve the peer review ecosystem.