📊 statistics

Rejoinder: The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review

This rejoinder addresses feedback on the ICML 2023 Ranking Experiment by structuring its response around four key themes: framing peer review as statistical estimation, mitigating equity and strategic issues in the Isotonic Mechanism, integrating complementary signals like reviewer rankings, and proposing a human-centered framework for the era of generative AI.

Original authors: Buxin Su, Jiayao Zhang, Natalie Collina, Yuling Yan, Didong Li, Kyunghyun Cho, Jianqing Fan, Aaron Roth, Weijie Su

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Buxin Su, Jiayao Zhang, Natalie Collina, Yuling Yan, Didong Li, Kyunghyun Cho, Jianqing Fan, Aaron Roth, Weijie Su

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of top-tier computer science research (specifically Artificial Intelligence) as a massive, chaotic music festival. Every year, thousands of bands (researchers) want to play on the main stage (get their papers published). But there are only a few hundred judges (reviewers) to listen to them, and the festival is growing so fast that the judges are drowning in applications.

The authors of this paper, a team of statisticians and computer scientists, argue that we need to stop treating this festival like a simple popularity contest and start treating it like a statistical puzzle. Here is a breakdown of their ideas using everyday analogies:

1. The Problem: Too Many Tickets, Not Enough Judges

The festival is exploding. In 2026, they received over 26,000 songs to judge. The judges are tired, and they can't listen to every song perfectly. Sometimes a judge is having a bad day, sometimes they just don't "get" a specific genre, and sometimes the music is just too good or too bad for them to judge fairly.

The authors say: "We can't just let the crowd decide (like counting 'likes' on social media) because the crowd isn't expert enough. We need a better way to figure out which songs are actually the best."

2. The Solution: The "Author's Playlist" (The Isotonic Mechanism)

The authors propose a clever trick called the Isotonic Mechanism.

The Analogy: Imagine you are a judge at a cooking contest. You have to rate 100 dishes. You might be a bit harsh on spicy food or a bit too soft on desserts. Your scores are "noisy" (a bit messy).
Now, imagine the chef (the author) who made the dish is standing right there. They know their own food better than anyone. They know exactly how their 10 dishes compare to each other.

The Old Way: The chef just says, "My dish is a 9 out of 10." (This is easy to lie about; everyone wants a 10).
The New Way: The chef is asked to rank their own 10 dishes from best to worst. "Dish A is better than Dish B, which is better than Dish C."

Why this works: It is very hard to lie about a ranking of your own work without getting caught. If you say your "Dish C" is the best, but your "Dish A" is actually terrible, the math can spot the lie. By asking authors to rank their own submissions, the system can "clean up" the messy scores from the judges. It's like using the chef's own knowledge to fix the judge's bad hearing.

3. Addressing the "Unfairness" Worry

Some people worried: "What if a famous chef with 50 dishes gets a huge advantage over a new chef with only 2 dishes? The ranking system works better for the person with more items to rank."

The Authors' Fix: They realized that if you just let one person rank everything, it's unfair. So, they suggest a "grouping" strategy.

The Analogy: Instead of asking the famous chef to rank all 50 dishes in one giant line, ask them to group them into small "tasting menus" (e.g., "The Soup Menu," "The Dessert Menu"). Rank the soups against each other, and the desserts against each other.
The Result: This levels the playing field. The math shows that even with this grouping, the system still makes the scores much more accurate, but it stops the famous chefs from dominating the results just because they have more entries.

4. The "Strategy" Problem: Can Chefs Cheat?

The authors admit that if the ranking actually decides who gets to play on the main stage, chefs might try to game the system.

The Risk: A chef might rank a terrible dish as "great" just to get it accepted, or rank their best dish lower to "save" it for next year.
The Reality Check: The authors say, "We can't stop all cheating, but we can design the rules to make cheating harder." They suggest starting small (using rankings just to flag which papers need a second look, rather than immediately accepting/rejecting them) so we can learn how people behave before making it a high-stakes game.

5. The Future: Humans + AI, Not AI vs. Humans

The paper ends with a big picture view. Artificial Intelligence is changing how research is written and reviewed. AI can write code, check math, and even draft reviews.

The Authors' Stance:

Don't replace the humans. AI is a tool, like a super-powered calculator. It can organize the data and find patterns, but it shouldn't be the final judge.
The "Human-in-the-Loop": Think of AI as the stagehand who sets up the lights and organizes the instruments. The human judge is the one who decides if the music is actually good.
Why? Because science isn't just about data; it's about creativity, ethics, and "taste." AI might miss the subtle soul of a paper or hallucinate (make things up). Humans need to stay in charge of the final decision.

Summary

The paper argues that to fix the overwhelmed system of judging AI research, we should treat it like a math problem. By asking authors to honestly rank their own work, we can "de-noise" the judges' scores and find the best papers more accurately. However, we must be careful not to let the system be gamed, and we must ensure that while AI helps us organize the chaos, humans remain the ones holding the gavel.

Technical Summary: The ICML 2023 Ranking Experiment

Problem Formulation
The paper frames the peer review process in Machine Learning (ML) and Artificial Intelligence (AI) conferences as a statistical estimation problem. With submission volumes growing exponentially (e.g., ICML 2026 receiving 26,600 valid submissions) and the supply of qualified reviewers failing to keep pace, the community faces a challenge in estimating the latent quality ( $\theta_i$ ) of tens of thousands of submissions based on noisy signals. Traditional review scores (e.g., 1–8 scales) are viewed as noisy estimators of this latent quality, influenced by reviewer bias, variance, and strategic behavior. The authors argue that while the "market" (e.g., GitHub stars) cannot replace formal peer review due to the need for reliable filtering of correctness and novelty, the estimation process itself must be improved to handle the scale and speed of modern AI research.

Methodology: The Isotonic Mechanism
The core contribution is the application of the Isotonic Mechanism, a method developed to elicit truthful private information from authors to denoise review scores.

Concept: The mechanism treats authors as strategic stakeholders who possess private knowledge about the relative quality of their own submissions. By asking authors to rank their own submissions, the system can leverage this internal comparative information.
Incentive Structure: The mechanism is designed such that, under specific constraints, authors are incentivized to report their true rankings. This shifts the focus from solely monitoring reviewers to incorporating author-side signals.
Integration: The calibrated scores derived from these rankings are used to adjust the raw review scores. In the ICML 2023 experiment, these adjusted scores were primarily used to direct Area Chair (AC) attention rather than to make automatic accept/reject decisions, serving as a low-stakes deployment to test efficacy.
Extensions: The paper discusses potential extensions, including:
- Topic-based Partitioning: To address concerns about heterogeneity in improvement (where prolific authors might benefit disproportionately), the authors propose partitioning an author's submissions into smaller, topic-based blocks. This allows the mechanism to be applied independently within blocks, smoothing out the influence of ranking length.
- Complementary Signals: The framework is designed to integrate with other signals, such as reviewer rankings, pairwise comparisons, structured metadata (e.g., theoretical vs. empirical), and self-predictions.

Key Results and Findings
Based on the ICML 2023 experiment and subsequent analysis:

Estimation Accuracy: The Isotonic Mechanism significantly improves the accuracy of quality estimation. The paper reports a reduction in Mean Squared Error (MSE) and Mean Absolute Error (MAE) compared to raw review scores.
Heterogeneity Mitigation: While initial concerns suggested that authors with many submissions would gain disproportionate calibration benefits, the analysis shows that the averaging effect across co-authors dilutes this influence. Furthermore, the proposed "topic-based partition" variant further equalizes the relative improvement across authors with different submission counts. For instance, under the partition variant, the regression slope of percentage improvement on submission count drops significantly (e.g., MSE slope from 1.97 to 1.24), indicating a more equitable distribution of gains.
Predictive Power: A follow-up study (Su et al., 2025a) indicates that authors' own rankings are powerful predictors of future citation counts.
Strategic Behavior: The paper acknowledges that in a high-stakes deployment (where scores directly influence acceptance), strategic behavior (e.g., ranking papers to maximize acceptance counts rather than truth) may emerge. However, the authors argue that low-stakes, randomized, or staggered deployments are necessary to study and mitigate these behaviors before full-scale implementation.

Significance and Claims
The paper positions itself not as a final solution to peer review, but as a step toward a human-centered reform of the scientific publication system in the AI era.

Statistical Perspective: It asserts that viewing peer review as an estimation problem allows statisticians to contribute rigorously to the field through mechanism design and noise reduction.
Human-in-the-Loop: The authors emphasize that the Isotonic Mechanism does not replace human judgment or reviewers. Instead, it integrates an additional human signal (author rankings) with existing review scores to improve the reliability of the estimation process.
AI Context: The paper argues that as AI tools increasingly assist in writing and reviewing, traditional cues of quality (e.g., polished language) may become less reliable. In this context, eliciting contextual human knowledge through incentive-aligned mechanisms becomes more valuable.
Future Agenda: The authors call for a broader research agenda involving the design of mechanisms that account for complex incentive structures, the integration of diverse signals (reviewer and author rankings), and the development of hybrid publication systems that balance rapid dissemination with post-publication auditing. They highlight that the intersection of mechanism design and statistical modeling represents a rich frontier for the statistics community.

The paper concludes by noting that while the ICML 2023 experiment was a voluntary, low-stakes test, a randomized experiment with consequential outcomes is underway at ICML 2026 to further validate these findings.