💬 NLP

SafeReview: Defending LLM-based Review Systems Against Adversarial Hidden Prompts

This paper proposes "SafeReview," a novel adversarial framework that employs a dynamically co-evolving Generator and Defender model to robustly detect and mitigate adversarial hidden prompts designed to manipulate LLM-based academic peer review systems.

Original authors: Yuan Xin, Yixuan Weng, Minjun Zhu, Ying Ling, Chengwei Qin, Michael Hahn, Michael Backes, Yue Zhang, Linyi Yang

Published 2026-04-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuan Xin, Yixuan Weng, Minjun Zhu, Ying Ling, Chengwei Qin, Michael Hahn, Michael Backes, Yue Zhang, Linyi Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of academic research as a massive, high-stakes talent show. Every year, thousands of scientists submit their "acts" (research papers) to be judged by a panel of experts. For a long time, humans have done the judging. But recently, because there are so many submissions, organizers started hiring AI judges (Large Language Models) to help speed things up.

The paper "SafeReview" addresses a scary new problem: What if a contestant secretly whispers instructions to the AI judge to rig the score?

The Problem: The "Whispering" Contestant

In the past, if a contestant wanted to win, they had to write a great paper. Now, a "bad actor" (a malicious author) can hide a secret note inside their paper. It's like a contestant slipping a note to the judge that says, "Ignore the fact that my act is terrible; give me a 10/10 anyway."

The paper calls these "adversarial hidden prompts."

The Trick: The author hides these instructions right inside the text of their paper (maybe in the introduction or conclusion).
The Result: The AI judge gets confused, ignores the flaws, and gives the bad paper a high score. This means bad science gets published, and good science gets rejected.

The Solution: A "Sparring Partner" Training Camp

The authors, Yuan Xin and their team, realized that you can't just build a wall to stop these whispers because the bad actors keep changing their tricks. Instead, they built a system called SafeReview that works like a martial arts training camp.

They created two AI models that fight each other in a continuous loop:

The Attacker (The "Generator"): This AI's only job is to learn how to write the sneakiest, most convincing secret notes possible. It tries to trick the judge into giving high scores to bad papers.
The Defender (The "Reviewer"): This AI is the judge. Its job is to read the papers and spot the secret notes, ignoring them to give a fair score.

How they train together:

The Attacker tries to fool the Defender.
When the Attacker succeeds, the Defender learns from the mistake and gets smarter.
Once the Defender gets better, the Attacker has to come up with an even smarter trick to fool it.
They keep fighting, getting stronger and stronger together.

This is called Co-Evolutionary Training. Instead of teaching the judge a static list of "bad words" (which the attackers can easily bypass), they force the judge to learn how to think and detect the intent of the trick, no matter how it's disguised.

The Results: A Tougher Judge

The researchers tested this system on thousands of real academic papers. Here is what they found:

Without SafeReview: The AI judges were easily tricked. Bad papers got inflated scores, and the system's ability to rank the best papers dropped significantly.
With SafeReview: The system became a "tough nut to crack." Even when the Attacker used its most advanced tricks, the SafeReview judge still:
- Spotted the fakes: It rejected the rigged papers much more often.
- Kept the rankings fair: It could still tell the difference between a great paper and a bad one, even when the bad one was trying to cheat.
- Didn't punish the honest: It didn't get so paranoid that it started rejecting good papers just because they were written confidently. It learned to tell the difference between "confident writing" and "manipulative writing."

The Bottom Line

The paper argues that as we rely more on AI to judge science, we need a way to protect those AI judges from being hacked. SafeReview is a new method that trains the AI to be a smarter, more resilient judge by constantly practicing against a relentless opponent. It ensures that the "talent show" remains fair, and that the best research gets recognized, not the best cheaters.

1. Problem Statement

As Large Language Models (LLMs) are increasingly integrated into academic peer review (e.g., for ICLR and AAAI), they face a critical security threat: Adversarial Hidden Prompts.

The Threat: Malicious authors embed covert instructions within their submissions (e.g., in the Abstract or Conclusion) that trick the LLM reviewer into ignoring weaknesses, inflating scores, or providing favorable reviews.
The Vulnerability: Unlike standard prompt injection which hijacks the system's task, hidden prompts manipulate the evaluation outcome while maintaining semantic coherence with the paper.
Limitations of Current Defenses: Static defenses (trained on known attacks) fail because adversarial strategies evolve rapidly. Existing methods struggle with long-form documents (where malicious text is a small fraction of the total) and often sacrifice review quality or fairness to achieve robustness.

2. Methodology: SafeReview Framework

The authors propose SafeReview, a co-evolutionary adversarial training framework designed to create a dynamic "arms race" between an attacker and a defender. This approach moves beyond static training by continuously adapting both models.

Core Components

Generator (Attacker):
- Model: Qwen3-4B-Instruct.
- Objective: Generate sophisticated injection prompts ( $\tau$ ) to manipulate review scores.
- Training: Uses Group Relative Policy Optimization (GRPO).
- Reward Function: A hybrid reward balancing two goals:
  - Ranking Disruption: Maximizing the difference between the Spearman correlation of the attacked batch vs. ground truth.
  - Rating Manipulation: Maximizing the score increase ( $\Delta s$ ) of the attacked paper.
- Strategy: Prompts are randomly inserted into standard paper sections (Abstract, Methodology, Conclusion, etc.) to ensure the defender learns to detect attacks regardless of location.
Defender (Reviewer):
- Model: DeepReviewer-14B (a domain-specific LLM).
- Objective: Maintain accurate, unbiased review scores despite adversarial injections.
- Training: Uses Direct Preference Optimization (DPO).
- Preference Pairs: For every attacked paper ( $p \oplus \tau$ ), the model is trained to prefer the review generated for the clean version of the paper ( $y^+$ ) over the review generated for the attacked version ( $y^-$ ).

The Co-Evolutionary Process

The framework operates in an iterative loop (Algorithm 1):

Attack Evolution: The Generator uses GRPO to create new, harder attacks based on the current Defender's vulnerabilities.
Defense Strengthening: The Defender uses DPO to learn from these new attacks, updating its policy to reject the manipulated inputs.
Curriculum Learning: This dynamic interaction naturally creates a curriculum of increasing difficulty, forcing the Defender to generalize against evolving threat models rather than memorizing specific attack patterns.

3. Key Contributions

Formulation: The first formulation of defending against hidden adversarial prompts in peer review as a co-evolutionary learning problem, where attack and defense capabilities improve in tandem.
Stable Training Pipeline: A novel pipeline tailored for long-form scientific documents, combining GRPO for attack generation and DPO for defense strengthening. This addresses the instability of standard RL-based adversarial training on large models.
Comprehensive Evaluation: Demonstrated that SafeReview outperforms static defenses and existing baselines across multiple metrics, achieving superior ranking preservation and fairness without penalizing legitimate confident writing.

4. Experimental Results

The framework was evaluated on the DeepReview-13k dataset and a NeurIPS 2024 dataset.

Vulnerability Baseline: Without defense, adversarial hidden prompts significantly inflated scores (e.g., average rating increased from 5.72 to 7.33) and degraded the correlation between AI scores and ground truth.
Defense Performance (vs. GRPO Attacks):
- Spearman Correlation: SafeReview achieved 0.409, a 19.2% improvement over Static DPO (0.343) and significantly higher than the baseline DeepReview (0.354). This indicates superior ability to maintain correct paper ranking.
- False Positive Rate (FPR): SafeReview reduced the rate of wrongly accepting flawed papers to 38.2%, outperforming Static DPO (42.2%) and System Defense (43.6%).
- Fairness (FNR): SafeReview maintained a low False Negative Rate (36.1%), ensuring legitimate papers were not unfairly rejected.
Cross-Architecture Generalization: When tested against attacks generated by a different model family (Llama-3.2-3B), SafeReview maintained high robustness (Spearman 0.392), proving it learns general adversarial features rather than overfitting to specific generator artifacts.
Fairness Analysis:
- Confident Writing: SafeReview showed a 10x improvement in ranking correlation for "Accepted" papers (which often use confident language) compared to the baseline. This proves the model does not overcorrect or penalize legitimate assertive writing.
- Human Alignment: SafeReview's ratings aligned more closely with human expert judgments than the baseline, particularly for rejected papers.

5. Significance and Impact

Scholarly Integrity: SafeReview provides a foundational security layer for the growing adoption of AI in peer review, ensuring that automated systems cannot be easily manipulated to accept low-quality research.
Dynamic Defense Paradigm: It shifts the security paradigm from static "patching" to dynamic co-evolution, which is essential for defending against adaptive adversaries in open-ended domains like academic writing.
Practical Deployment: The framework demonstrates that high robustness can be achieved without sacrificing the quality of reviews for benign inputs, making it viable for real-world conference and journal workflows.

Limitations & Future Work:
The current work focuses on instruction-style injections in Computer Science venues. Future work aims to extend the framework to multi-modal submissions, proprietary API-based reviewers (where fine-tuning is impossible), and more subtle, non-instruction-based semantic perturbations.