Re2: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussions

To address limitations in existing peer review datasets such as low diversity and inconsistency, the paper introduces Re², the largest consistency-ensured dataset comprising initial submissions, reviews, and multi-turn rebuttal discussions from 24 conferences to support the development of effective LLM assistants for authors and reviewers.

The Gist

Imagine the world of scientific research as a massive, bustling Grand Library. Every day, thousands of new books (research papers) are written and dropped off at the front desk, hoping to be added to the permanent collection.

To get into the collection, these books must pass a strict Gatekeeper System called "Peer Review." Here's how it usually works:

1. The Author submits their book.
2. The Reviewers (experts who volunteer their time) read it and write a report: "This is great!" or "This needs work."
3. The Author gets the report, fixes the book, and writes a Rebuttal (a letter saying, "I fixed it! Here's why you were wrong about that one thing").
4. The Reviewer reads the rebuttal, maybe changes their mind, and gives a final verdict.

The Problem: The Library is Overwhelmed

Right now, the library is drowning. Too many books are being submitted, and there aren't enough Gatekeepers to read them.

• The Bottleneck: Reviewers are exhausted, leading to slower decisions and lower quality checks.
• The Cycle of Failure: Many authors submit books that aren't ready yet because they don't have a good way to check their own work before handing it in. They get rejected, fix it, and resubmit the same book multiple times, wasting everyone's time.

Scientists have tried to use AI (Large Language Models) to help. They want the AI to act like a "Pre-Check" for authors or a "Helper" for reviewers. But there was a big problem: The AI was being trained on bad recipes.

The "Bad Recipe" Problem

Imagine you are trying to teach a cooking student how to make a perfect cake.

• Old Datasets: You gave them a recipe that said, "Take the cake after the student already fixed the burnt parts and added extra frosting." The student learns to judge the finished cake, not the raw batter. When they try to judge a new, raw cake, they get confused because the data didn't match the reality of the initial submission.
• The Missing Step: Most old datasets also ignored the "Rebuttal" phase. They didn't teach the AI how to handle the back-and-forth argument between the author and the reviewer. It was like teaching a lawyer to write a closing statement but never letting them practice cross-examination.

The Solution: Introducing "Re2"

The authors of this paper built Re2, which is like a massive, perfectly organized training kitchen for AI.

Here is what makes Re2 special, using simple analogies:

1. The "Fresh Batter" Rule (Consistency)

In the old days, datasets mixed up "raw batter" (initial submissions) with "finished cakes" (revised versions). Re2 is strict: Every single paper in this dataset is the "Fresh Batter" version.

• Why it matters: It ensures the AI learns to judge a paper exactly as a human reviewer sees it on day one. There is no confusion about which version is being discussed.

2. The "Giant Mosaic" (Diversity)

Old datasets were like a puzzle with only 5 pieces from one specific box (mostly just one conference, ICLR). Re2 is a giant mosaic made of 19,926 papers from 45 different conferences and workshops.

• Why it matters: The AI learns that "good writing" looks different in a medical journal than in a computer science journal. It becomes a true generalist, not just an expert in one niche.

3. The "Conversation Simulator" (Multi-turn Rebuttal)

This is the biggest innovation. Re2 doesn't just show the Reviewer's note and the Author's reply as two separate documents. It stitches them together into a continuous, multi-turn conversation, like a chat log.

• The Analogy: Imagine training a customer service bot. Old datasets gave it a script of "Complaint -> Response." Re2 gives it the entire phone call, including the pauses, the "Wait, I meant this..." moments, and the final agreement.
• Why it matters: This allows AI to learn how to argue, negotiate, and clarify just like a real human reviewer. It can now act as a dynamic assistant that helps authors refine their work before they even submit it.

What Did They Do With It?

They took some standard AI models (like LLaMA and Qwen) and fed them this "Re2" training data.

• Before Training: The AI was like a polite but clueless intern who would say "Great job!" to everything just to be nice.
• After Training: The AI became a sharp, critical editor. It could predict if a paper would be accepted, guess the score, write a review that sounded human, and even engage in a back-and-forth debate about the paper's flaws.

The Bottom Line

Re2 is the first time we have a massive, high-quality, "real-world" dataset that captures the entire lifecycle of a scientific paper—from the first draft to the final argument.

By using this dataset, we can build AI tools that:

1. Help Authors: Act as a "pre-submission coach" to fix mistakes early, so fewer bad papers get submitted.
2. Help Reviewers: Act as a "co-pilot" to draft reviews and handle rebuttals, reducing the crushing workload on human volunteers.

In short, Re2 is the training manual that finally teaches AI how to be a real, helpful member of the scientific community, rather than just a fancy text generator.

Technical Summary

1. Problem Statement

The peer review system in fields like AI and Computer Science is facing a crisis due to a surge in submission volumes, leading to reviewer shortages and declining review quality. A significant contributor to this overload is the repeated resubmission of substandard manuscripts, caused by authors' lack of effective tools for self-evaluation prior to submission.

While Large Language Models (LLMs) show promise in assisting both authors and reviewers, their performance is currently bottlenecked by the quality and limitations of existing peer review datasets. The paper identifies three critical gaps in current datasets (e.g., PeerRead, MReD, NLPeer):

1. Limited Data Diversity: Most datasets rely heavily on a single conference (typically ICLR) due to data accessibility issues, lacking broad representation across different venues.
2. Data Inconsistency (Version Mismatch): Many existing datasets utilize revised versions of papers rather than initial submissions. This creates a logical inconsistency where the paper content fed into a model does not match the version the reviewers actually evaluated, undermining the training of review-generation and prediction models.
3. Insufficient Support for Interaction: Existing datasets largely focus on static review generation or score prediction. They fail to capture the rebuttal and discussion stages, which are critical multi-turn conversations between authors and reviewers. Current datasets either lack this data or handle it in a coarse, unstructured manner.

2. Methodology

The authors introduce Re2, a comprehensive dataset designed to address the above limitations through rigorous data collection and processing pipelines.

Data Collection and Scope

• Source: Data was crawled from OpenReview using the official API.
• Scale: The dataset covers 24 conferences and 21 workshops spanning from 2017 to 2025 (45 venues total).
• Components:
  • Re2-Review: Contains 19,926 initial paper submissions and 70,668 review comments.
  • Re2-Rebuttal: Contains 14,830 initial submissions paired with 53,818 rebuttals and discussion threads.

Data Processing and Consistency Assurance

• Initial Submission Verification: To solve the version mismatch problem, the authors collected submission deadlines for each conference/year. They used web scraping to extract the latest version uploaded before the deadline from the "Revision History" of each paper, ensuring the text matches the version reviewed.
• Format Unification:
  • PDF to Text: Used Doc2X (a commercial tool superior for math formulas) to convert PDFs into LaTeX and Markdown.
  • Custom Extraction: Developed customized extraction logic for each conference-year pair to handle diverse review formats (metadata, scores, confidence levels, aspect ratings).
• Multi-turn Conversation Construction (Rebuttal Stage):
  • Fragment Merging: Authors often post multiple consecutive responses due to character limits. The authors concatenated these into single turns.
  • Noise Removal: A hybrid (manual + automated) strategy was used to identify and remove "reminder" or "urging" posts that do not constitute substantive rebuttal content.
  • Global Response Handling: When authors provided a "Global Response" addressing multiple reviewers, these were inserted into the dialogue as referential context rather than direct replies, preserving logical flow while ensuring all relevant information is available to the model.
  • Structure: The final output is a structured, self-consistent multi-turn conversation dataset.

3. Key Contributions

1. Largest Consistency-Ensured Dataset: Re2 is the largest real-world peer review dataset to date, covering the widest range of conferences and the most complete review stages (initial submission, reviews, ratings, rebuttals, discussions, score changes, meta-reviews, and final decisions). Crucially, it guarantees that all paper contents are initial submissions.
2. Multi-turn Conversation Paradigm: It is the first dataset to frame the rebuttal and discussion stages as a structured multi-turn conversation, enabling the training of dynamic, interactive LLM assistants rather than just static text generators.
3. Comprehensive Evaluation: The authors provide statistical analysis of the dataset and demonstrate its effectiveness through four downstream tasks: Acceptance Prediction, Score Prediction, Review Generation, and Rebuttal-Discussion Conversation.

4. Experimental Results

The authors fine-tuned open-source LLMs (LLaMA-3.1-8B and Qwen2.5-7B) on Re2 and compared them against zero-shot baselines and specialized peer-review models (e.g., SEA-E, DeepReviewer).

• Acceptance & Score Prediction:
  • Fine-tuned models showed significant improvements over zero-shot versions.
  • Zero-shot LLMs tended to "please" users by accepting all papers (high recall, low precision). After fine-tuning on Re2, models became more discriminative, with MAE and MSE for score prediction decreasing significantly (e.g., MAE dropped from ~1.96 to ~1.14 for LLaMA-3.1-8B).
• Review Generation:
  • Fine-tuned LLaMA-3.1-8B achieved the highest scores in BLEU and ROUGE-L, outperforming specialized baselines.
  • In semantic similarity (EmbedCos), fine-tuned models showed massive gains (e.g., +49.9% improvement for LLaMA-3.1-8B), indicating better alignment with the language structure of real peer reviews.
• Rebuttal-Discussion Conversation:
  • Fine-tuned models consistently outperformed zero-shot versions across lexical, semantic, and LLM-as-a-Judge metrics (Quality, Constructiveness, Accuracy, Completeness, Clarity).
  • This confirms that Re2 effectively teaches models how to engage in the nuanced, multi-turn dialogue required for academic rebuttals.

5. Significance and Impact

• Alleviating Reviewer Burden: By enabling LLMs to generate high-quality, constructive reviews and simulate rebuttal discussions, Re2 supports the development of tools that can assist reviewers, potentially mitigating the shortage of human reviewers.
• Author Self-Evaluation: The dataset facilitates the creation of pre-submission self-evaluation tools. Authors can use these models to identify weaknesses in their initial drafts before submission, reducing the rate of substandard resubmissions.
• Research Foundation: Re2 sets a new standard for data quality in the field of automated peer review by prioritizing consistency (initial versions) and interaction (multi-turn dialogue), paving the way for more advanced, interactive AI research assistants in academia.

Limitations: The current dataset and experiments focus on textual and tabular data, excluding visual elements (figures/tables), though the raw data includes them. Future work aims to integrate vision-language models.