🤖 machine learning

Metag: A dataset to build agentic meta-reviewing capabilities

This paper introduces Metag, a publicly available dataset comprising 349 high-quality action items that align reviewer concerns, author resolutions, and manuscript diffs to facilitate the development of AI agents capable of automating and enhancing the meta-review process by tracking changes made during the peer review and rebuttal stages.

Original authors: Anirudh Sundar, Min Chen, Divya Tadimeti, Gemma Zhang, Alice Li, Nigel Boachie Kumankumah, Pavan Uttej Ravva, Sadid Hasan, Somya Chatterjee, Pruthvi Prakash Navada, Xiao Wang, Yue Kang, Sulaiman Vesal

Published 2026-08-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Anirudh Sundar, Min Chen, Divya Tadimeti, Gemma Zhang, Alice Li, Nigel Boachie Kumankumah, Pavan Uttej Ravva, Sadid Hasan, Somya Chatterjee, Pruthvi Prakash Navada, Xiao Wang, Yue Kang, Sulaiman Vesal, Larry Heck

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of scientific research, the journey from a first draft to a published paper is rarely a straight line. It is a conversation. When a researcher submits a study to a major conference, it is read by several experts who point out flaws, ask for more data, or suggest clearer explanations. The author then enters a "rebuttal" phase, a back-and-forth dialogue where they promise to fix these issues. Finally, a senior editor, known as a meta-reviewer, must read all these comments and promises to decide if the paper is ready for publication. This final step is a heavy burden. The meta-reviewer must verify that the author actually made the changes they promised, often by hunting through hundreds of pages of text to find where a specific sentence was altered or a new chart was added. As the number of scientific papers grows, this manual checking becomes increasingly difficult, risking that important promises go unkept or that genuine improvements are overlooked.

A team of researchers at Microsoft and the Georgia Institute of Technology has tackled this problem by creating a new tool designed to teach computers how to do this verification work. They built a dataset called Metag, which acts as a training ground for artificial intelligence agents. The goal is to create a system that can automatically read a reviewer's concern, look at the author's response, and then instantly locate the exact spots in the revised paper where those changes happened. To build this, the researchers gathered hundreds of real-world examples from a major machine learning conference. They took the original versions of accepted papers and compared them to the final, polished versions. Using software, they generated a detailed list of every single difference between the two documents, from a changed word to a moved paragraph. They then asked human experts to link these specific changes to the promises made in the review discussions. The result is a collection of 349 high-quality examples where a reviewer's request is directly tied to the specific edits an author made to satisfy it.

The researchers used this dataset to test how well different types of artificial intelligence could perform this linking task. They tried simple methods that look for matching words, as well as advanced language models that can understand context and nuance. The results showed that while computers can find some changes, the task is surprisingly difficult. The best-performing models managed to correctly identify the relevant changes about 40% of the time. This might sound low, but in this specific field, it represents a significant step forward. The study found that simple word-matching often fails because authors rarely use the exact same words in their changes as they did in their promises; they might say they "clarified a point" and then simply rewrite a paragraph without using the word "clarify." The most successful models were those capable of understanding the intent behind the request rather than just hunting for keywords.

Despite these successes, the paper makes it clear that the problem is not yet solved. The best models still miss many relevant changes and sometimes flag edits that have nothing to do with the reviewer's request. The researchers note that their work is limited to papers from a single conference and relies on finding the original versions of papers on public archives, which is not always possible. They also found that human annotators did not always agree on which changes were relevant, suggesting that the task itself involves a degree of subjectivity. However, the creation of Metag provides a crucial foundation. By offering a clear, structured way to measure progress, this dataset allows other scientists to build better tools. The ultimate vision is a future where AI acts as a helpful assistant to human editors, highlighting the specific pages and paragraphs where authors have kept their promises, thereby making the peer-review process more transparent, efficient, and trustworthy.

Technical Summary: Metag – A Dataset for Agentic Meta-Reviewing

Problem Statement
The scientific peer-review process faces increasing pressure due to the growth in conference submissions. While AI tools have been developed to assist authors in writing and reviewers in identifying errors, the meta-reviewing process remains largely under-explored. Meta-reviewers must synthesize vast amounts of information from reviews, author rebuttals, and manuscript revisions to determine if authors have adequately addressed reviewer concerns. A critical bottleneck is traceability: the ability to connect specific reviewer action items (concerns and author commitments) to the concrete textual changes made in the final manuscript. Existing document comparison tools can identify differences between PDF versions but fail to explain which reviewer comment motivated a specific change, hindering transparency and accountability.

Methodology
The paper introduces Metag, a dataset designed to bridge the gap between reviewer-author dialogues and observable manuscript revisions. The dataset construction follows a six-stage pipeline:

Data Acquisition: The authors scraped submissions from ICLR 2024 via OpenReview. They identified accepted papers and extracted the full reviewer-author dialogue threads (reviews, comments, and rebuttals).
PDF Retrieval: To capture the "before" and "after" states of the papers, the authors linked OpenReview submissions to their pre-review arXiv versions using Semantic Scholar. They ensured the retrieved arXiv version was the closest to the submission date, excluding post-submission updates.
Diff Computation: Using PyMuPDF and Git's histogram diff algorithm, the system computed structured differences between the pre-review and camera-ready PDFs. This generated a list of edits (insertions, deletions, replacements) tagged with text spans, page numbers, and surrounding context.
Action Item Extraction: An LLM (Gemma-3-27B-IT) was prompted to extract discrete "action items" from the dialogue. These items are pairs consisting of a reviewer concern and an author's explicit commitment to revise the manuscript (e.g., "We revised Section 3...").
Human Annotation: Annotators used a custom Python-based interface to view side-by-side PDFs with color-coded diff overlays. Their task was to link the extracted action items to the specific diffs in the all_diffs list that implemented the change.
Agreement and Assembly: Each action item was annotated by two independent annotators. The final dataset retained only the diffs found in the intersection of both annotators' selections to ensure high quality.

The resulting dataset contains 349 high-quality action items tied to 323,786 total diffs across 137 papers. Each instance includes the action item, the full list of candidate diffs, and binary labels indicating relevance.

Key Contributions

Metag Dataset: The first dataset explicitly linking reviewer action items from scientific dialogues to specific, structured document diffs. Unlike prior work (e.g., ARIES) that aligns comments to text spans, Metag aligns them to computed diffs between document versions, preserving insertion/deletion/move operations and page locations.
Annotation Framework: A novel tool for side-by-side PDF diff-linking that allows annotators to navigate, search, and select relevant changes efficiently.
Benchmarking: The paper establishes a baseline for the "diff classification" task, evaluating both lexical retrieval methods (BM25, TF-IDF) and various Large Language Models (LLMs) including Gemma-3-27B-IT, GPT-5.6-Sol, DeepSeek-V4-Pro, and Kimi-K2.5.

Results
The authors evaluated the difficulty of the task by testing multiple models on the validation and test splits:

Lexical Baselines: Traditional retrieval methods (BM25, TF-IDF) performed poorly, achieving micro-F1 scores around 0.10. This suggests that relevant revisions often cannot be identified by direct lexical overlap alone, as action items describe intent rather than repeating the revised text.
LLM Performance:
- GPT-5.6-Sol achieved the strongest and most stable performance, reaching a test micro-F1 of 0.360 and macro-F1 of 0.398. It demonstrated a better balance of precision and recall compared to other models.
- DeepSeek-V4-Pro showed high variance, achieving a high validation micro-F1 (0.410) but dropping to 0.261 on the test set.
- Kimi-K2.5 achieved high recall but lower precision.
- Gemma-3-27B-IT variants (zero-shot and LoRA) underperformed compared to hosted models, with zero-shot models exhibiting severe over-selection (predicting hundreds of irrelevant diffs).
Inter-Annotator Agreement: Human annotators agreed on the exact set of diffs 37% of the time, with a mean Jaccard similarity of 50.1%. Binary agreement on whether any diff existed for an action item yielded a Cohen's $\kappa$ of 0.369, indicating fair agreement beyond chance and highlighting the inherent subjectivity in linking dialogue to specific edits.

Significance and Claims
The paper claims that Metag enables the development of agentic human-in-the-loop systems to assist meta-reviewers. By automating the traceability between review dialogue and manuscript evidence, such systems can:

Verify whether promised revisions were implemented.
Distinguish substantive changes from unrelated edits.
Inspect the manuscript evidence underlying an author's response.

The authors position Metag as a foundational resource for improving transparency and traceability in peer review. They explicitly note limitations, including the dataset's derivation from a single conference (ICLR 2024) and the lack of access to intermediate revision versions during the review process, which restricts the granularity of the analysis. The work is presented as a step toward building tools that empower meta-reviewers to navigate revisions more efficiently, rather than a fully solved problem for automated meta-reviewing.

Technical Summary: Metag – A Dataset for Agentic Meta-Reviewing

More like this