💻 computer science

How Sensitive Are Safety Benchmarks to Judge Configuration Choices?

This paper demonstrates that the configuration of LLM judges, particularly prompt wording, is a substantial and previously under-examined source of measurement variance in safety benchmarks, causing significant shifts in harmful-response rates and instability in model safety rankings.

Original authors: Xinran Zhang

Published 2026-04-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xinran Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher grading a stack of student essays. You have a strict rubric (a set of rules) for what counts as a "bad" essay. Now, imagine you have to grade these essays using a very smart, but slightly mood-dependent, robot assistant.

This paper is about a surprising discovery: The way you ask the robot to grade the essays matters just as much as the robot itself.

Here is the breakdown of the study using simple analogies:

1. The Setup: The "Robot Grader" Experiment

The researchers wanted to see how safe different AI models are. To do this, they used a standard test called HarmBench, which contains 400 tricky questions (like "How do I make a bomb?" or "How do I steal a copyright?").

They asked six different AI models to answer these questions. Then, they used a single "Judge AI" (a specific version of Claude Sonnet) to read the answers and decide: Is this answer harmful or safe?

Usually, scientists treat the "Judge AI" and the instructions given to it as a fixed, unchangeable tool. They assume if you change the wording of the instructions slightly, the result should stay the same.

The researchers asked: What if the wording does change the result?

2. The Experiment: 12 Different "Scripts"

The researchers created 12 different versions of the instructions (prompts) for the Judge AI. They changed two main things:

The Structure: Did they ask the judge to look at every tiny detail one by one (like a detective checking clues), or to look at the whole answer as a big picture (like a principal reading a report)?
The Persona: Did they tell the judge, "You are a strict, senior safety expert," or did they just say, "Please evaluate this"?

They also wrote three slightly different versions of the instructions for each style (e.g., "You are a safety expert" vs. "You are an AI safety specialist").

3. The Shocking Result: The "Mood Swing" of the Judge

When they ran the test, they found that the wording of the instructions completely changed the scores.

The Analogy: Imagine you ask a robot to rate how "spicy" a soup is.
- If you say, "You are a professional chef who hates spice," it might rate the soup as "Mild."
- If you say, "You are a health inspector who is very strict about safety," it might rate the exact same soup as "Dangerously Hot."
- Even if you just change "Chef" to "Culinary Expert," the score might jump up or down significantly.

The Numbers:

For one of the AI models being tested, the "harmful" rate jumped from 13% to 37% just by changing the prompt wording. That is a 24-point swing.
Even when they kept the main instructions the same and only changed a few words (surface rewording), the scores still swung by 20 points.
This means that if you run the same safety test twice with slightly different instructions, you could get two completely different conclusions about how safe an AI is.

4. Who Got Hurt the Most? (The Categories)

Not all types of questions were affected equally:

Copyright Questions: These were the most sensitive. The "harmful" rate swung by nearly 40 points depending on the prompt. It seems the judge struggled to decide if copying text was "bad" or "okay" based on how it was asked.
Harassment Questions: These were the most stable. The score didn't change at all (0 points). Everyone agreed that harassment was bad, no matter how the instructions were phrased.

5. The Ranking Chaos

Because the scores changed so much, the ranking of the AI models also got messy.

Imagine a race where the winner changes every time you change the color of the starting line.
The researchers found that for the "middle-tier" AI models, their safety ranking flipped back and forth constantly. One prompt said Model A was safer than Model B; a slightly different prompt said Model B was safer.
The "worst" and "best" models stayed in their spots, but the middle ones were impossible to rank reliably.

6. The Takeaway

The paper concludes that safety benchmarks are currently "under-specified."

Think of it like a thermometer that gives different temperatures depending on whether you hold it with your left hand or your right hand. The researchers aren't saying the thermometer is broken, but they are saying: We can't trust a single reading.

They argue that when scientists report how safe an AI is, they shouldn't just give one number based on one specific set of instructions. They need to realize that the "instructions" are a huge part of the experiment, not just a boring detail. If you change the words, you change the result.

In short: The way you ask the question is just as important as the question itself. If you want to know if an AI is safe, you can't just ask it once with one set of words; the answer depends entirely on how you phrase the request.

1. Problem Statement

Large Language Models (LLMs) are increasingly evaluated for safety using automated "LLM-as-Judge" systems (e.g., HarmBench). In these benchmarks, a target model generates responses to harmful prompts, and a separate LLM judge classifies them as "harmful" or "safe."

The Assumption: Current safety benchmarks typically treat the judge configuration (the specific combination of the judge model and the judge prompt) as a fixed, neutral implementation detail.
The Gap: While there is evidence that LLM behavior is sensitive to minor instruction changes in training and QA evaluation, the sensitivity of safety evaluation metrics to specific judge prompt wording has not been systematically studied.
Core Question: How much do safety metrics (harmful-response rates) and model rankings vary based solely on changes to the judge prompt, holding the judge model constant?

2. Methodology

The authors conducted a controlled factorial experiment to isolate the effects of prompt wording.

Experimental Design: A $2 \times 2 \times 3$ $2 \times 2 \times 3$ factorial design resulting in 12 distinct judge prompt variants.
- Axis 1: Structure.
  - Atomic: The judge decomposes the response into individual claims and aggregates them (harmful if any claim is dangerous).
  - Holistic: The judge applies a rubric to the response as a whole without decomposition.
- Axis 2: Framing.
  - Identity: The system prompt assigns an expert persona (e.g., "You are a senior AI safety evaluator...").
  - Neutral: The system prompt uses task-oriented language without a persona (e.g., "Evaluate AI model responses...").
- Axis 3: Surface Wording. For each of the 4 conditions above, 3 independently worded variants were created (same structure/framing, different phrasing).
Judge Model (Primary Analysis): All 12 prompts were applied using a single judge model (Claude Sonnet 4-6) to eliminate the confound of different judge models.
Target Models & Data:
- 6 Target Models: Claude Sonnet 4-6, GPT-4.1, DeepSeek V3, Kimi K2.5, MiniMax M2, GLM-5.
- 400 Behaviors: Standard HarmBench text set covering 7 categories (Chemical/Biological, Copyright, Cybercrime, Harassment, Harmful/General, Illegal, Misinformation).
- Total Judgments: 28,812 judgments (after filtering parse errors).
Supplementary Analysis: A multi-judge experiment using three different models (Sonnet, MiniMax M2, GLM-5) to assess if judge-model choice adds further variance.

3. Key Contributions

Quantification of Prompt Sensitivity: Demonstrated that prompt wording alone shifts measured harmful-response rates by up to 24.2 percentage points (pp) on a single judge model.
Surface Rewording Impact: Showed that even "equivalent" prompts (same structure/framing, different surface wording) cause swings of up to 20.1 pp.
Category-Specific Sensitivity: Identified that sensitivity varies drastically by harm category, ranging from 39.6 pp for Copyright to 0 pp for Harassment.
Ranking Instability: Found that model safety rankings are moderately unstable, with mid-tier models frequently swapping positions depending on the prompt used.
Open Resources: Released all 12 judge prompt variants and analysis code.

4. Key Results

A. Magnitude of Variance

Overall Swing: For DeepSeek V3, the harmful rate ranged from 13.3% to 37.5% across the 12 prompts (a 24.2 pp swing). Four out of six models exceeded a 13 pp swing.
Source of Variance: The majority of the variance came from surface rewording within the same condition (mean 10.8 pp) rather than the structural or framing axes themselves. This implies that "equivalent" prompts are not functionally equivalent in safety evaluation.
Multi-Judge Comparison: Changing the judge model added variance, but the variance introduced by prompt wording alone was comparable to the variance introduced by switching judge models.

B. Ranking Stability

Kendall's $\tau$ : The mean agreement between model rankings induced by different prompts was 0.89 (range 0.80–0.96).
Reversals: While extreme models (safest and least safe) remained stable, mid-tier models showed high instability.
- Example: The ranking between GLM-5 and Sonnet flipped in 83.6% of bootstrap resamples.
- Example: Kimi and Sonnet flipped in 67.9% of cases.

C. Category Sensitivity

Most Sensitive: Copyright (39.6 pp swing). The authors attribute this to the interpretive ambiguity of copyright violations (e.g., paraphrasing vs. direct copying), which makes the outcome highly dependent on how the judge is instructed to interpret "harm."
Most Robust: Harassment (0 pp swing). The authors suggest this may be due to ceiling effects or the clear binary nature of harassment definitions.

D. Prompt Design Insights

Structure/Framing Effects: While Holistic prompts yielded slightly higher harmful rates than Atomic, and Neutral framing yielded slightly higher rates than Identity, these differences were small (~~2–3 pp) compared to the variance caused by surface wording (~~10+ pp).
The "v3" Anomaly: Prompts labeled "v3" consistently produced higher harmful rates than v1 or v2, despite semantic equivalence. This suggests that specific phrasing (e.g., "seasoned content safety specialist" vs. "senior AI safety evaluator") induces different levels of strictness.

5. Significance and Implications

Under-Specification of Benchmarks: Current safety benchmarks are under-specified. A single "headline" safety metric is unreliable because it can shift by >20 pp based on arbitrary prompt choices.
Reproducibility Crisis: Benchmarks must specify the exact judge prompt text, not just the model used. "Equivalent" prompts yield different results.
Evaluation Protocol:
- Safety evaluations should report the range of metrics across multiple prompt wordings rather than a single point estimate.
- Prompt wording should be treated as a first-class experimental variable, not a fixed implementation detail.
- Special caution is needed when interpreting results for Copyright categories due to extreme sensitivity.
Future Directions: The paper calls for human annotation of dispute subsets to determine which prompt configuration is "most accurate" and suggests exploring prompt ensembling to reduce noise.

Conclusion

The paper fundamentally challenges the reliability of current LLM safety benchmarks. It proves that the judge prompt is a substantial, previously under-examined source of measurement variance. As the field moves toward using these benchmarks for deployment decisions, the community must characterize and report this variance to ensure safety evaluations are robust and reproducible.