🤖 AI

Refute-or-Promote: An Adversarial Stage-Gated Multi-Agent Review Methodology for High-Precision LLM-Assisted Defect Discovery

This paper introduces "Refute-or-Promote," an adversarial multi-agent framework that employs stratified context hunting, cross-model critique, and mandatory empirical validation to filter out false positives in LLM-assisted defect discovery, achieving high precision and real-world impact including multiple CVEs and standard acceptances across security libraries and compilers.

Original authors: Abhinav Agarwal

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Abhinav Agarwal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the editor of a massive newspaper. Every day, thousands of people send in tips about crimes they think they've spotted in your city. Most of these tips are actually just misunderstandings, pranks, or people seeing things that aren't there. If you try to investigate every single tip, your newsroom would collapse, and you'd stop believing the few real crimes that do get reported.

This is exactly the problem Large Language Models (LLMs) face when they try to find software bugs (defects). They are incredibly good at sounding confident and making things sound plausible, but they are terrible at knowing if something is actually true. They often "hallucinate" fake bugs, overwhelming human experts with noise.

The paper "Refute-or-Promote" proposes a new way to fix this. Instead of asking the AI to just "find bugs," it sets up a high-stakes courtroom drama to filter out the fakes.

Here is how the system works, broken down into simple steps:

1. The Setup: The "Detective Squad"

First, the system sends out a team of "hunters" (AI agents) to look for potential bugs in a piece of software.

The Trick: These hunters don't all look at the same thing in the same way. Some look at recent changes, some look at old mistakes, and some look at specific parts of the code. This is called Stratified Context Hunting. It's like sending three different detectives to the same crime scene: one looks for footprints, one looks for fingerprints, and one looks for the weather report.

2. The Core Mechanism: The "Devil's Advocate" Court

Once a hunter finds a "suspect" (a potential bug), it doesn't get published immediately. It enters a Stage-Gated Courtroom.

The Prosecution (Adversarial Agents): A team of AI agents is assigned a single, ruthless job: Prove the suspect is innocent. They are not allowed to say "maybe." They must find a reason why the bug doesn't exist. They are the "Kill Mandate" team.
The Defense (Creative Agents): Another team tries to prove the bug is real.
The Rule: If the Prosecution can find even one solid reason why the bug is fake, the case is killed immediately. The bug is thrown out. The system only moves forward if the Prosecution fails to disprove it.

3. The "Fresh Eyes" Test (Cross-Model Critic)

Even if the first court agrees a bug is real, there's a danger: all the AI agents might have learned the same wrong facts from their training data. They might all agree on a lie because they all read the same bad textbook.

To fix this, the system brings in a Cross-Model Critic. This is an AI from a completely different "family" (a different company's model) that hasn't seen the previous arguments. It looks at the case with "cold eyes." If this outsider says, "Wait, this doesn't make sense," the bug gets killed. This catches errors that the whole first team missed because they were all thinking alike.

4. The Final Gate: The "Lab Test"

This is the most important part. The paper describes a massive failure where 80+ AI agents unanimously agreed a specific security flaw existed. They were all so confident they wrote reports about it.

The Reality: The flaw didn't exist.
The Solution: The system realized that "AI consensus" isn't proof. The final gate requires Empirical Validation. You have to actually run a test in a computer lab to see if the bug happens. If you can't make the bug happen in real life, it doesn't matter how smart the AI thinks it is. The case is closed.

The Results: Why This Matters

The authors ran this system for a month on real-world software (like the code that runs your browser or secure encryption).

The Filter: The system started with about 171 potential bugs.
The Cleanup: The "Prosecution" and "Lab Tests" killed about 79% of them because they were fake or unprovable.
The Success: The remaining 21% were real. They resulted in:
- 4 new security vulnerabilities being officially reported (CVEs).
- Fixes for the C++ programming language standard.
- Merged code fixes for major software libraries.

The Big Lesson

The paper teaches us a profound lesson about AI: Agreement does not equal truth.

If 80 AI agents all say "The sky is green," they are probably all wrong because they are all trained on the same data. But if you have one agent trying to prove them wrong, and you force them to run a real-world test, you find the truth.

Refute-or-Promote isn't about making AI smarter; it's about building a system that is skeptical by design. It forces the AI to try to break its own ideas before it's allowed to share them with humans. It turns the AI from a "confident guesser" into a "rigorous scientist."

1. Problem Statement

Large Language Models (LLMs) show promise in identifying software vulnerabilities but suffer from a critical precision crisis. Their tendency to prioritize "plausibility" over "correctness" leads to an overwhelming volume of false positives (plausible-but-wrong reports).

Consequences: This influx acts as a denial-of-service attack on open-source maintainers, degrading the credibility of genuine findings. Notable examples include the permanent closure of curl's bug bounty program and the pausing of HackerOne's Internet Bug Bounty program in March 2026 due to AI-amplified submission volumes.
The Core Challenge: Standard cooperative debate or single-agent self-critique often fails because LLMs share training data biases, leading to correlated errors (e.g., multiple agents unanimously agreeing on a non-existent vulnerability).

2. Methodology: Refute-or-Promote

The authors propose Refute-or-Promote, an inference-time reliability pattern designed to filter false positives through a four-stage adversarial pipeline. Unlike cooperative debate, this system uses hard adversarial kill mandates and context asymmetry.

Core Architectural Components

Stratified Context Hunting (SCH):
- Candidate Generation: Multiple "hunter" agents generate candidates in parallel, stratified by:
  - Source: Prior defects, git hotspots, spec text.
  - Scope: Non-overlapping subsystems (e.g., parsing vs. memory safety).
  - Wave: Iterative re-seeding based on learnings from previous kills/promotions.
- Self-Critique: Hunters must articulate why their own candidate might not be exploitable before reporting.
Adversarial Kill Mandates (Stages A & B):
- Parallel Tracks: Each stage runs a Creative Track (arguing for exploitability) and an Adversarial Track (arguing against it).
- Context Asymmetry: Adversarial agents are given only the claim, not the creative agent's reasoning, to prevent "anchoring" on the advocate's framing.
- Cold-Start Review: Some agents receive minimal context to catch reasoning errors that informed agents miss.
- Kill Condition: A candidate is killed if any adversarial agent produces a code-grounded refutation.
Cross-Model Critic (CMC) (Stage D):
- Agents from a different model family (e.g., using a different underlying architecture/training data than the hunters) perform an independent critique.
- Goal: To detect correlated training-data errors that same-family review misses (analogous to N-version programming).
Mandatory Empirical Gate (Stage C):
- No candidate proceeds to disclosure without empirical validation (PoC execution, runtime testing, or compiler divergence checks).
- CVSS Calibration: Adversarial agents systematically adjust severity scores downward to prevent over-claiming.

3. Key Contributions

The Refute-or-Promote Protocol: A codified, four-stage adversarial pipeline combining SCH, kill mandates, cross-model verification, and empirical gating.
Real-World Validation: The methodology was tested over a 31-day campaign across 7 targets (security libraries, ISO C++ standard, compilers), resulting in 36+ validated outcomes.
Quantitative Analysis:
- Kill Rate: ~79% aggregate retrospective kill rate; 83% prospective kill rate on a consolidated protocol subset ( $n=30$ ).
- Severity Calibration: Adversarial review adjusted severity scores downward in 8 of 9 measured cases.
Honest Failure Reporting: The paper documents critical failures, including a case where 80+ agents unanimously endorsed a non-existent Bleichenbacher padding oracle in OpenSSL, which was only killed by a single empirical test. This highlighted the necessity of the empirical gate.

4. Results & Outcomes

The pipeline produced diverse, externally validated artifacts across three evidentiary states:

Public CVEs: 4 CVEs assigned (3 public, 1 embargoed), including CVE-2026-33150 and CVE-2026-33179 in libfuse, and CVE-2026-34183 in OpenSSL.
Standards & Compiler Fixes:
- LWG 4549 accepted into the ISO C++ Working Paper.
- 5 merged ISO C++ editorial PRs.
- 3 compiler conformance bugs (GCC/MSVC).
- RFC 9000 errata filed.
Security Fixes: 8 merged security-related fixes without CVEs (e.g., in OpenSSL and wolfSSL).
Cost Efficiency: Approximately $62 per CVE at subscription pricing (excluding researcher labor), demonstrating high efficiency compared to manual auditing.

Notable Case Studies:

OpenSSL (Bleichenbacher Failure): 80+ agents confirmed a fake vulnerability. A single empirical test disproved it. This failure drove the mandatory inclusion of Stage C (Empirical Gate).
libfuse (Cross-Model Success): A Cross-Model Critic (using a different model family) found 3 bugs and 3 correctness issues in proposed fixes that the same-family review missed.
lcms2 (Unanimous Kill Resurrection): A candidate unanimously killed by agents (due to a perceived lack of delivery path) was resurrected by a human orchestrator who found a smaller trigger, proving that "unanimity" can be a false negative signal as well.

5. Significance & Implications

Precision Over Coverage: The paper argues that for LLM-assisted security, precision (filtering false positives) is more critical than coverage. The system is designed to filter, not just find.
Unanimity is a Low-Signal Event: The study demonstrates that unanimous agreement among LLMs often reflects shared training biases rather than truth. Context asymmetry and cross-model verification are essential to break these correlated error loops.
Human-in-the-Loop Necessity: While the pipeline automates the filtering of false positives, it does not eliminate the need for humans. Humans are essential for:
- Resurrecting true positives that were unanimously killed (false negatives).
- Target selection and severity negotiation.
- Handling "over-refusal" by lower-tier models in authorized security workflows.
Generalizability: The methodology was successfully transferred to non-security domains, solving 5 previously unsolved instances on SWE-bench Verified (SymPy) and one hard task on SWE-rebench, suggesting the "adversarial kill mandate" pattern is a general reliability primitive for LLMs.

Conclusion

Refute-or-Promote establishes that adversarial, stage-gated multi-agent review with mandatory empirical validation can effectively solve the "plausibility crisis" in LLM-assisted defect discovery. By forcing agents to disprove findings rather than just debate them, and by introducing cross-model and cross-context isolation, the system achieves high-precision defect discovery suitable for responsible disclosure, moving beyond benchmark performance to real-world impact.