🤖 machine learning

SafetyRepro: Configuration-Conditional Rank Instability on Alignment Benchmarks

The paper "SafetyRepro" demonstrates that configuration choices in alignment benchmarks can fundamentally reverse pairwise safety rankings, introducing a theoretical framework and evaluation protocol to quantify and address this rank instability.

Original authors: Yanhang Li, Zhichao Fan, Zexin Zhuang

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yanhang Li, Zhichao Fan, Zexin Zhuang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to judge which of three runners (let's call them Runner A, Runner B, and Runner C) is the fastest. You want a fair race to see who wins.

In the world of AI, these "runners" are Large Language Models (LLMs), and the "race" is a benchmark test designed to see which one is safer, more truthful, or less biased.

This paper, SafetyRepro, argues that the results of these races are often fake because the person running the race (the evaluator) has too much control over the rules, the track, and the stopwatch. In fact, by simply changing the rules slightly, the evaluator can make any runner win, even if they are all running the same race.

Here is the breakdown of the paper's findings using simple analogies:

1. The "Rulebook" Problem

When a benchmark paper says "Model A is safer than Model B," it sounds like a scientific fact. But the paper argues this is more like a game of Rock-Paper-Scissors where the person holding the paper can secretly change the rules before the game starts.

Before the AI even answers a question, the evaluator has to choose:

The Prompt Template: How the question is phrased (e.g., "Answer this" vs. "You are a helpful assistant, answer this").
The Decoding: How the AI guesses the next word (e.g., being very strict vs. being a bit creative).
The Scoring: How the answer is graded (e.g., looking for a specific letter like "A" vs. reading a whole paragraph).

The paper tested three popular AI models on five famous safety benchmarks. They didn't just run the test once; they ran it through 612 different combinations of these rules (like trying every possible combination of shoes, socks, and laces).

2. The "Flip" (The Main Discovery)

The most shocking finding is what the authors call the "Rank Flip."

Imagine you have a leaderboard.

Scenario 1: You use "Rule Set A." The leaderboard says: Runner A > Runner B > Runner C.
Scenario 2: You switch to "Rule Set B" (a different prompt or scoring method). Suddenly, the leaderboard says: Runner C > Runner A > Runner B.

The paper found that on every single benchmark they tested, the evaluator could flip the winner just by changing the configuration.

On one specific test called XSTest (which checks if an AI refuses to answer dangerous questions), the evaluator could make any of the 6 possible rankings happen. They could make the worst model look like the best, and the best look like the worst, simply by tweaking the settings.

The Metaphor: It's like a judge in a cooking competition who can decide that "spiciness" is the only thing that matters. If they do, a mild soup loses to a spicy curry. But if they decide "sweetness" is the only thing that matters, the curry loses to a cake. The paper shows that AI benchmarks are currently like this: the "winner" depends entirely on which flavor the judge decides to taste first.

3. The "Black Box" of Different Tools

The paper also looked at what happens if you use three different "racing software" packages (like lm-evaluation-harness, HELM, and Inspect AI) to test the same model with the same settings.

The Result: The scores were wildly different.

On one test, the scores varied by 21.7 percentage points just because a different software package was used.
Why? Because even though they were testing the same "task," the software packages were actually measuring slightly different things. One might be checking if the AI picked the right letter, while another checks if the AI wrote a sentence that contains the right letter.

The Metaphor: It's like three different people measuring the height of a building. One measures from the ground to the roof. One measures from the basement to the roof. One measures from the roof to the sky. They are all "measuring height," but they get completely different numbers because their tools and definitions don't match.

4. What This Means for "Safety"

The paper concludes that when you read a headline saying "Model X is safer than Model Y," you should be very skeptical.

The Claim: The paper does not say the models are bad. It says the ranking is unstable.
The Reality: The statement "Model A is safer" is actually a statement about the pair of (Model A + The Specific Rules Used). If you change the rules, the statement might become false.
The Solution Proposed: The authors suggest a "GRID card." Just like a nutrition label on food tells you exactly what ingredients are in it, a benchmark score should come with a label listing every single rule used to generate that score. Without that label, the score is meaningless.

Summary in One Sentence

The paper proves that current AI safety tests are so sensitive to tiny changes in how they are run that an evaluator can essentially "rig" the results to make any AI model look like the winner or the loser, meaning we cannot trust current rankings until we standardize the rules and disclose exactly how the test was run.

Technical Summary: SafetyRepro: Configuration-Conditional Rank Instability on Alignment Benchmarks

Problem Statement
The paper addresses a critical gap in the reproducibility of Large Language Model (LLM) alignment benchmarks. While pairwise comparisons (e.g., "Model A is safer than Model B") are treated as quantitative verdicts in the literature, they rely on evaluation harnesses that under-specify critical configuration choices. Evaluators must select prompt templates, decoding settings, few-shot levels, scoring rules, and quantization schemes. The authors argue that these choices are often plausible but under-specified in benchmark papers, leading to a lack of reproducibility where the same model pair can yield contradictory rankings depending on the harness configuration.

Methodology
The authors construct a "commit-stamped" evaluation protocol to test the stability of pairwise rankings across a finite configuration envelope.

Experimental Grid: The study evaluates three 7–9B open-weight instruction-tuned models (Qwen-2.5-7B-Instruct, Mistral-7B-Instruct-v0.3, and Yi-1.5-9B-Chat) across five alignment-related benchmarks: TruthfulQA, BBQ, ToxiGen, CrowS-Pairs, and XSTest.
Configuration Envelope: A 612-cell grid is swept, varying four axes per (model, benchmark) pair:
1. Prompt Template: Four levels (Minimal, Role-framed, Detailed, Chain-of-Thought).
2. Decoding: Three levels (Greedy, T=0.3, T=0.7).
3. Few-shot: Three levels (0, 3, 5 shots).
4. Scoring: Two levels (Free-form regex extraction vs. Logprob argmax).
Constraints: The grid is fixed to NF4 quantization (simulating 16 GB consumer hardware) and a fixed item subset per benchmark to isolate harness variance from data-subsampling variance.
Metrics:
- Operator-Controllable Pairwise-Disagreement Rate ( $\rho_{flip}$ ): Defined as $\min(N_+, N_-) / |C_b|$ , where $N_+$ and $N_-$ are the counts of configurations where Model A > Model B and Model B > Model A, respectively. This metric measures the existence of strict reversals within the envelope.
- Ranking Concordance ( $\tau_R$ ): Average Kendall $\tau$ over induced 3-model rankings.
- Variance Attribution: A four-way Type-II ANOVA to partition variance between model identity and implementation axes (template, decoding, few-shot).
Case Study: A separate analysis re-runs a single nominal anchor configuration (Qwen2.5-7B-Instruct) across three mainstream evaluation packages (lm-evaluation-harness, HELM-lite, Inspect AI) to demonstrate implementation non-equivalence.

Key Findings

Universal Rank Instability: On every benchmark tested, configuration choice alone is sufficient to flip the pairwise verdict between models. The metric $\rho_{flip}$ is non-zero for all benchmarks, indicating that strict ordering is not identifiable within the tested envelope.
Maximum Instability:
- XSTest: The most volatile benchmark; all six possible orderings of the three models are reachable within the full envelope. Even in a "core tier" (excluding stress-tier CoT and diverse decoding), 5 of 6 orderings remain reachable.
- TruthfulQA: Shows the highest pairwise disagreement rate ( $\rho_{flip} \approx 47.9\%$ on the full envelope, $\approx 40.7\%$ on the core tier).
- BBQ and ToxiGen: Also exhibit significant instability, with $\rho_{flip}$ values around 43.8% and 35.4% respectively.
Implementation vs. Model Dominance: Variance partitioning reveals that implementation main effects explain as much or more aggregate variance than model identity on three of the four free-form benchmarks (BBQ, ToxiGen, XSTest). Only TruthfulQA is model-dominated. Interaction effects are large (41–54%), suggesting the partition is a qualitative direction rather than a precise ratio.
Scoring Path Sensitivity: Switching between free-form regex parsing and logprob argmax scoring produces materially different scores (up to 22.7 percentage points on BBQ and 21.7 on TruthfulQA in the cross-package study), confirming that different scoring paths measure non-equivalent candidate objects.
Cross-Package Non-Equivalence: Running the same nominal configuration across three different evaluation packages yields score spreads of up to 22.7 percentage points. This is attributed to implementation non-equivalence (different candidate objects, prompt boilerplate, and ground-truth mappings) rather than a single-axis package effect.

Contributions

Metric: The introduction of the operator-controllable pairwise-disagreement rate ( $\rho_{flip}$ ), a configuration-conditional metric that quantifies the existence of strict rank reversals within a defined practice-derived envelope.
Data: A releasable 612-cell evaluation grid covering 15 (model, benchmark) pairs with commit-stamped configuration rules.
Evidence: A case study demonstrating implementation non-equivalence across three major evaluation packages, showing that disclosing a configuration is insufficient for score reproducibility across different harnesses.
Tooling: A research-level disclosure template (GRID card) to record harness configurations alongside scores, acknowledging that while it improves transparency, it does not solve the problem of claimants gaming configuration neighborhoods.
Robustness Checks: Analysis of a conservative-core sub-envelope (4 configs) and a cross-family scale probe (7B vs. 32B/34B) which rules out single-family confounds but does not claim a general scaling law.

Significance and Claims
The paper claims that a benchmark score ranking Model A above Model B is a property of the (model, harness) pair, not the models alone. The authors assert that without disclosing both the model and the specific harness configuration, such rankings cannot be reproduced.

The significance lies in closing a "theory–benchmark loop" by providing a one-line test (Proposition 1) for whether a pairwise ranking is free of strict reversal on a given configuration grid. The authors emphasize that their findings are bounded: they do not claim to have measured "LLM-evaluation reproducibility" as a population, nor do they claim to have solved reproducibility. Instead, they demonstrate that within a deliberately bounded, practice-derived envelope (consumer hardware, specific model sizes, fixed item subsets), the "strict ordering" primitive fails to identify a stable truth. The paper concludes that certifying which published rankings are identified requires neither new theory nor new compute, but rather the application of this configuration-conditional test.

1. The "Rulebook" Problem

2. The "Flip" (The Main Discovery)

3. The "Black Box" of Different Tools

4. What This Means for "Safety"

Summary in One Sentence

More like this