Automatic Paper Reviewing with Heterogeneous Graph Reasoning over LLM-Simulated Reviewer-Author Debates

The paper proposes ReViewGraph, a novel framework that simulates multi-round reviewer-author debates using LLM-based agents, encodes their argumentative interactions into a heterogeneous graph, and applies graph neural networks to achieve more accurate and reasoned paper reviews, outperforming existing baselines by 15.73%.

The Gist

Imagine you are the editor of a very busy magazine. Every week, hundreds of people send you their stories. You need to decide which ones to publish, but you only have a few hours to read them all. You ask three friends (your reviewers) to read them and give you their opinions.

Sometimes, your friends disagree. One says, "This story is brilliant!" Another says, "The ending makes no sense!" A third says, "I like the characters, but the grammar is terrible."

The Problem with Current AI Reviewers
Right now, if you ask a standard AI (like a basic chatbot) to help you decide, it usually does one of two things:

1. The "Quick Reader": It reads the story once and gives a quick guess. It often misses the subtle arguments or gets confused by complex details.
2. The "Debate Club": It tries to simulate a conversation between the friends. But often, this conversation is messy. The AI might just list what everyone said without really understanding how they are arguing with each other. It might miss that Friend A and Friend B actually agree on the grammar, even if they disagree on the plot.

The Solution: ReViewGraph
The authors of this paper built a new tool called ReViewGraph. Think of it as a "Smart Debate Map."

Here is how it works, using a simple analogy:

1. The Simulation (The Rehearsal)

Instead of just reading the paper once, ReViewGraph sets up a role-playing game with AI agents.

• It creates three "Reviewer" bots and one "Author" bot.
• The Reviewers read the paper and write their critiques.
• The Author reads the critiques and writes a response (a "rebuttal"), saying, "Oh, I see your point, I will fix that," or "Actually, I disagree because..."
• The Reviewers read the Author's response and update their opinions.

This creates a rich, multi-round conversation, just like a real academic review process.

2. The Map (The Graph)

This is the magic part. Instead of just reading the text of the conversation, ReViewGraph draws a map of the arguments.

Imagine a subway map:

• The Stations (Nodes):
  • One station is the Paper Title.
  • Other stations are Categories like "Is the math new?" (Methodological Novelty) or "Is the writing clear?" (Writing Fluency).
  • Then there are stations for every single Opinion the reviewers and author had.
• The Tracks (Edges):
  • The tracks connect the stations, but they have colors and labels.
  • A Green Track might mean "Agree." (Reviewer 1 and Reviewer 2 both think the math is great).
  • A Red Track might mean "Disagree." (Reviewer 1 thinks the math is bad; Reviewer 2 thinks it's good).
  • A Yellow Track might mean "Clarify." (The Author explained a confusing point to the Reviewer).
  • A Blue Track might mean "Compromise." (The Author agreed to add more experiments).

3. The Reasoning (The Detective)

Once the map is built, ReViewGraph uses a special kind of "brain" (a Graph Neural Network) to look at the whole map at once.

It doesn't just count how many people said "Yes" or "No." It looks at the structure:

• Scenario A: Three reviewers say "No," but they all agree with each other, and the author couldn't fix their concerns. The map shows a strong cluster of "Red Tracks." The AI says: "Reject."
• Scenario B: Two reviewers say "No," but they are arguing with each other (one says "Too simple," the other says "Too complex"), and the third reviewer says "Yes." The Author successfully clarified the confusion. The map shows a mix of "Red" and "Green" tracks that resolve into a "Yes." The AI says: "Accept."

Why is this better?

• It sees the big picture: It understands that an argument isn't just a sentence; it's a relationship between people and ideas.
• It avoids "Hallucinations": Because it builds a structured map of facts and opinions, it's less likely to make things up or get confused.
• It's fair: It treats the "Author's defense" and the "Reviewers' critiques" as equal parts of the map, rather than just ignoring the author's voice.

The Result

The researchers tested this on real papers from a major computer science conference (ICLR). They found that ReViewGraph was much better at predicting whether a paper would be accepted or rejected than previous AI methods. It got about 15% better at making the right call.

In short: ReViewGraph turns a messy pile of reviews and responses into a clear, color-coded map of arguments, allowing the AI to "see" the truth of the debate rather than just guessing based on keywords.

Technical Summary

Here is a detailed technical summary of the paper "Automatic Paper Reviewing with Heterogeneous Graph Reasoning over LLM-Simulated Reviewer-Author Debates" (ReViewGraph).

1. Problem Statement

Automatic paper reviewing aims to alleviate the burden on human reviewers and improve the consistency of peer review. However, existing methods face significant limitations:

• Prompt-based approaches: Relying solely on Large Language Models (LLMs) with in-context prompting often leads to superficial reviews, hallucinations, and a lack of deep reasoning. These models struggle to capture the nuanced, multi-turn argumentative dynamics inherent in real peer review.
• Fine-tuned approaches: While they align better with expert criteria, they suffer from data scarcity, potential bias in training sets, and often fail to model the interactive negotiation between reviewers and authors.
• Lack of Structural Reasoning: Current methods generally treat reviews as isolated text generation tasks or simple classification problems, failing to explicitly model the complex relationships (agreement, disagreement, clarification, compromise) between multiple reviewers and the author.

2. Methodology: The ReViewGraph Framework

ReViewGraph proposes a novel framework that simulates reviewer-author debates and performs heterogeneous graph reasoning over the resulting interactions. The framework consists of three main stages:

A. Multi-Agent Reviewer-Author Debate Simulation

Instead of a single LLM pass, ReViewGraph employs a multi-agent collaboration framework to simulate a realistic peer review process:

1. Initial Review: Three regular reviewer agents (powered by multimodal LLMs) analyze the paper, identifying strengths and weaknesses across specific dimensions.
2. Author Rebuttal: An author agent generates point-by-point responses to the reviewers' feedback, clarifying misunderstandings and defending contributions.
3. Re-evaluation: Reviewers re-examine their initial judgments based on the author's rebuttal, refining their opinions or maintaining their stance.
   This process generates a rich, multi-turn dialogue containing diverse perspectives and argumentative structures.

B. Heterogeneous Debate Graph Construction

The simulated dialogue is transformed into a structured Heterogeneous Information Network (HIN) defined as $G = \{V, E, A, R\}$:

• Node Types ($A$):
  • Title: Represents the paper itself.
  • Evaluation Dimension: Critical criteria (e.g., Methodological Novelty, Experimental Completeness, Motivation Clarity, Writing Fluency).
  • Reviewer Opinion: Individual comments/arguments from reviewers.
  • Author Opinion: Individual responses from the author.
• Edge Types ($R$):
  • Paper-Dimension: Links the paper to evaluation criteria.
  • Dimension-Opinion: Links specific opinions to the criteria they address.
  • Inter-Reviewer Relations: Captures interactions between reviewers (e.g., agree, disagree, complement, progressive).
  • Reviewer-Author Relations: Captures interactions between reviewers and the author (e.g., accept, reject, clarify, compromise).

The system uses in-context prompting to extract opinion triplets and classify them into these specific relation types and dimensions, creating a semantically rich graph.

C. Heterogeneous Graph Reasoning

A Heterogeneous Graph Transformer (HGT) is applied to the constructed graph to perform relational reasoning:

• Heterogeneous Mutual Attention: Computes attention scores between nodes based on their types and the specific relation type connecting them, allowing the model to distinguish the semantic importance of different interaction types (e.g., a "disagree" edge vs. an "agree" edge).
• Message Passing & Aggregation: Information is passed between nodes, aggregated, and updated through multiple layers to generate contextualized node representations.
• Prediction: Node embeddings are pooled by type, concatenated, and fed into a feedforward neural network to predict the final decision (Accept/Reject).

3. Key Contributions

1. Novel Framework: Introduction of ReViewGraph, the first framework to model automatic reviewing as a heterogeneous graph reasoning task over LLM-simulated multi-round debates.
2. Structured Representation: Design of a semantically typed heterogeneous graph that explicitly captures fine-grained argumentative relations (e.g., compromise, clarification) and multi-perspective dynamics, moving beyond unstructured text.
3. Performance & Efficiency: The method achieves state-of-the-art performance without requiring parameter updates (fine-tuning) of the underlying LLMs, relying instead on structural reasoning and in-context prompting.

4. Experimental Results

• Datasets: Evaluated on three benchmark datasets derived from ICLR 2023, 2024, and 2025 submissions (OpenReview).
• Baselines: Compared against seven strong baselines, including:
  • Prompt-based: ICL, CoT, AI-Scientist.
  • Fine-tuned: CycleReviewer, DeepReview.
  • Graph-based: GraphEval.
• Performance: ReViewGraph consistently outperformed all baselines across Accuracy, Macro Precision, Macro Recall, and Macro F1.
  • Key Metric: Achieved an average relative improvement of 15.73% in Macro F1 over the second-best baseline (CycleReviewer-70B) on the ICLR 2025 dataset.
  • Statistical Significance: Improvements were confirmed via T-tests ($p < 0.05$).
• Ablation Studies: Removing key components (Title nodes, Evaluation Dimensions, Reviewer-Author edges, or Heterogeneity) significantly degraded performance, validating the necessity of the full graph structure.
• Case Studies: Demonstrated the model's ability to correctly identify "negative consensus" despite polite language (leading to correct Rejection) and "isolated dissent" amidst majority agreement (leading to correct Acceptance), tasks where other methods failed.

5. Significance

ReViewGraph represents a paradigm shift in automatic peer review by treating the review process not as a text generation task, but as a structured reasoning problem. By explicitly modeling the discourse-level interactions and argumentative dynamics between reviewers and authors, the framework:

• Enhances Interpretability: The graph structure provides a clear trace of how the final decision was reached based on specific arguments and relations.
• Improves Robustness: It mitigates LLM hallucinations and biases by grounding decisions in a structured debate simulation.
• Scalability: It offers a scalable solution that does not require expensive fine-tuning of large models, making it adaptable to various domains and evolving review criteria.

This work suggests that future AI-assisted scientific evaluation should focus on structured interaction modeling rather than just improving the raw generation capabilities of LLMs.