💻 computer science

When Scanners Lie: Evaluator Instability in LLM Red-Teaming

This paper reveals that automated LLM vulnerability scanners suffer from significant measurement instability due to unreliable evaluator components, and proposes a two-phase, reliability-aware framework that quantifies evaluator disagreement and validates assessments to substantially improve the accuracy and consistency of security risk reporting.

Original authors: Lidor Erez, Omer Hofman, Tamir Nizri, Roman Vainshtein

Published 2026-03-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lidor Erez, Omer Hofman, Tamir Nizri, Roman Vainshtein

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a security guard trying to test how good a new bank vault (an AI model) is at keeping out thieves. You send in a team of "red team" testers (attackers) to try and break in.

The problem isn't just whether the vault breaks; it's who gets to decide if the vault actually broke.

This paper, titled "When Scanners Lie," argues that the tools we use to measure AI security are often unreliable because the "judges" they use are inconsistent. Here is the breakdown using simple analogies:

1. The Core Problem: The "Referee" Problem

In the world of AI security, we use automated scanners to see how often an AI can be tricked into doing something bad. The scanner sends a tricky question (an attack) and gets an answer. Then, a "Judge" (an evaluator) looks at the answer and says, "Did the AI fail? Yes or No?"

The paper found that the score changes depending on who the referee is.

The Analogy: Imagine a soccer game.
- Referee A is a robot that only looks for the word "goal." If the ball crosses the line, it counts.
- Referee B is a human who watches the whole play. They see that the ball went out of bounds before crossing the line, so they say, "No goal."
- The Result: The same play happens, but Referee A says "Goal!" and Referee B says "No Goal." If you only listen to Referee A, you think the team is amazing. If you listen to Referee B, you think they are terrible.

The paper shows that most AI scanners use "Referee A" (simple, rule-based judges) or "Referee B" (complex AI judges), and they often disagree wildly. Sometimes, changing the judge changes the security score by as much as 33%.

2. The Discovery: "The Unstable Scoreboard"

The researchers tested a popular scanner called Garak. They kept the "thieves" (attacks) and the "vault" (AI model) exactly the same, but they swapped out the "referees."

The Shock: In 22 out of 25 types of attacks, the referees couldn't agree.
The Extreme: For some attacks, the referees disagreed almost 100% of the time. One said the AI was safe; the other said it was completely broken.
The Takeaway: The "Attack Success Rate" (ASR) isn't a fixed number like the temperature outside. It's a measurement that depends entirely on who is doing the measuring.

3. The Solution: A Two-Step "Double-Check" System

Instead of just picking one referee and hoping for the best, the authors propose a new framework with two phases:

Phase 1: The "Disagreement Alarm"

First, they run the test with two different referees at the same time.

The Analogy: Imagine you have two judges. If they both agree the player scored, great. But if they argue, you hit a red alarm button.
What it does: It flags the specific types of attacks where the referees are confused. It tells you, "Hey, for this specific type of trick, we don't know who is right yet."

Phase 2: The "Super-Referee" (Verification)

For the flagged, confusing attacks, they bring in a third, highly intelligent "Super-Referee" (a powerful AI with a special checklist) to make the final call.

The Analogy: If the two regular referees argue, you call in the Head Coach (the verifier) to watch the replay and make the final decision.
The Benefit: This doesn't require hiring thousands of humans (which is too slow and expensive). It uses a smart AI to check the work of the other AIs.

4. The Result: Smarter, Cheaper, Safer

By using this system, the researchers achieved three things:

Higher Accuracy: They boosted the accuracy of the security scores from 72% to 89%.
Cost Control: They didn't use the expensive "Super-Referee" for every test. They only used it when the regular referees disagreed. This is like only calling the Head Coach for controversial plays, not for every single kick.
Realistic Uncertainty: They realized that sometimes, even the best referees can't agree. So, instead of giving a single number (e.g., "10% vulnerability"), they now give a range (e.g., "The vulnerability is between 10% and 15%"). This is much more honest.

Summary

The paper is a wake-up call for AI security. It says: "Stop trusting the scoreboard blindly."

If you want to know if an AI is safe, you can't just run a scanner and look at the final number. You have to ask: Who was the referee? Did they agree with the other referees? And if they disagreed, did we have a better judge check it?

Without this "double-check" system, we might think an AI is safe when it's actually dangerous, or vice versa. The authors provide a practical toolkit to fix this, ensuring that our security measurements are actually measuring the AI's behavior, not just the referee's mood.

1. Problem Statement

Automated Large Language Model (LLM) vulnerability scanners are widely used to assess AI security by calculating the Attack Success Rate (ASR). However, the validity of ASR is compromised by a critical, often overlooked component: the evaluator (the mechanism determining if an attack succeeded).

The Core Issue: ASR is not an inherent property of a model but is operationally defined by the evaluator's decision rules.
Instability: Changing the evaluator (e.g., from a static keyword matcher to an LLM-based judge) while keeping the attack prompts and model outputs constant can drastically alter the reported ASR.
Consequence: This leads to measurement instability where vulnerability scores are sensitive to the choice of evaluator rather than the actual robustness of the target model. Current scanners often assume ASR is invariant to evaluator design, creating a methodological risk of misleading security assessments.

2. Methodology: A Two-Phase Reliability-Aware Framework

The authors propose a framework to diagnose and mitigate evaluator-induced instability without requiring extensive human annotation.

Phase I: Diagnostic (Evaluator Disagreement Analysis)

Objective: Quantify the instability caused by substituting evaluators.
Mechanism: Two different evaluators ( $E_1$ and $E_2$ ) are applied to the same set of prompt-response pairs.
Metric: The Disagreement Rate ( $D$ ) is calculated as the proportion of samples where the evaluators assign different binary success labels.
$D = \frac{1}{N} \sum_{i=1}^{N} \mathbb{1}\{E_1(x_i, y_i) \neq E_2(x_i, y_i)\}$
Action: Attack categories with a disagreement rate exceeding a threshold ( $\tau$ ) are flagged as "unstable." This identifies specific attack types where the reported ASR is unreliable.

Phase II: Remediation (Verification-Backed Evaluation)

Objective: Improve reliability for flagged attack categories.
Mechanism: An independent LLM-based Verifier is introduced. Unlike the initial evaluators, this verifier uses a reasoning-capable LLM with a structured system prompt to decompose the decision into multiple checks before assigning a final label.
Process:
1. The verifier re-evaluates the same prompt-response pairs used in Phase I.
2. It provides an independent reference signal to estimate the accuracy of the original evaluators.
3. Targeted Replacement: Practitioners can selectively replace unstable evaluators with more reliable ones (e.g., dynamic LLM judges) only for specific attack categories where the reliability gain justifies the computational cost.

3. Experimental Setup

Scanner: Garak (v0.13.2), a popular open-source LLM vulnerability scanner.
Scope: 25 distinct attack categories (e.g., Prompt Injection, Malware Generation, DAN).
Target Models: Three models of varying sizes and providers (Mistral-Small, CommandA, GPT-5-mini).
Evaluators Compared:
- Static: Garak's default rule-based evaluators (82% of the suite; regex, keyword matching).
- Dynamic: A generic LLM-based judge (Grok-4.1).
- Verifier: An independent LLM (GPT-5.2) used as the ground-truth proxy.

4. Key Results

Widespread Instability:
- 22 out of 25 attack categories (88%) exhibited evaluator disagreement rates exceeding the reliability threshold ( $\tau = 0.05$ ).
- 6 categories showed disagreement rates $> 0.50$ , meaning the choice of evaluator flipped the majority of success/failure decisions.
- Example: The "Misleading" attack had a disagreement rate of 0.97, while "Continuation" was stable at 0.013.
Accuracy Improvement:
- Replacing unstable static evaluators with dynamic ones (validated by the verifier) increased overall scanner accuracy from 72% to 89%.
- However, dynamic evaluators were not universally superior; in 4 of the 22 flagged categories, the static evaluator was more accurate.
ASR Uncertainty:
- Reported vulnerability scores can vary by ±33% depending on the evaluator used.
- Figure 5 in the paper illustrates significant uncertainty intervals for specific attack types (e.g., "Snowball" and "Leak Replay").
Cost-Benefit Trade-off:
- Selective Deployment: Replacing only the top high-gain evaluators increased accuracy to 81.9% with a negligible cost increase ($0.15).
- Diminishing Returns: Replacing all evaluators with dynamic ones actually reduced overall accuracy (dropping to 89.3% from a peak of 89.9%) because some static rules were better suited for specific attacks.
- Total Cost: Full replacement increased scan costs by ~$5.25 per scan.

5. Key Contributions

Demonstration of Evaluator-Dependent Instability: The paper empirically proves that ASR is a function of the evaluator design, not just the target model, challenging the assumption that vulnerability metrics are invariant.
Evaluator Disagreement Diagnostics: A novel technique to identify unreliable attack categories by measuring sample-level disagreement between evaluators, serving as a filter for scanner reliability.
Verification-Based Reliability Estimation: A scalable mechanism using an independent LLM verifier to estimate evaluator accuracy and guide targeted upgrades, avoiding the high cost of large-scale human annotation.

6. Significance and Implications

Reframing Red-Teaming: The work shifts the perspective of automated red-teaming from a simple "attack generation" task to a measurement reliability problem.
Practical Guidance for Security Teams:
- Do not trust raw ASR: Security practitioners must treat ASR scores with caution, acknowledging they are specific to the evaluation pipeline used.
- Targeted Upgrades: Instead of replacing all evaluators with expensive LLM judges, organizations should use the diagnostic framework to selectively upgrade only the unstable components, optimizing the balance between accuracy and cost.
- Ambiguity Detection: High disagreement between all evaluators (including the verifier) suggests the attack definition itself may be ambiguous and requires refinement.
Future of AI Safety: As automated red-teaming scales, ensuring the reliability of the underlying evaluators is critical for producing security assessments that accurately reflect model behavior.

In conclusion, the paper argues that evaluator design is a critical bottleneck in LLM security. By implementing a reliability-aware framework, organizations can significantly reduce measurement errors and make more informed decisions regarding model deployment and security postures.