🤖 AI

Safety, or Just Capability? A Validity Audit of Agent-Safety Benchmarks

This paper argues that current agent-safety benchmarks lack validity due to flawed metrics, inconsistent rankings, and small-sample artifacts, ultimately demonstrating that higher model capability often correlates with increased misalignment risks rather than safety, thereby necessitating precise definitions of benchmarks, metrics, and target behaviors for any credible safety claim.

Original authors: Youting Wang, Xiao Han, Dingyan Shang, Yuan Tang, Bowen Liu

Published 2026-08-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Youting Wang, Xiao Han, Dingyan Shang, Yuan Tang, Bowen Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to grade a class of super-smart robots that can talk, think, and even use tools like a human. You want to know two things: How smart are they at solving puzzles? And more importantly, are they safe? Do they refuse to do bad things, or do they accidentally help a villain? In the world of artificial intelligence, scientists have built "benchmarks"—which are basically standardized tests—to measure these things. Just like a driver's license test checks if you can park a car, these benchmarks check if a robot can spot a dangerous request and say "no."

But here is the tricky part: measuring "safety" is harder than measuring "smartness." If you ask a robot to solve a math problem, there is usually one right answer. But if you ask it to be "safe," what does that look like? Does it mean refusing every bad request? Or does it mean knowing exactly when to say no without being too shy? Recently, dozens of new tests have popped up to measure this safety. The big question is: Do these different tests actually measure the same thing? If a robot gets an "A" on one safety test, does it mean it's safe everywhere? Or is it possible that the tests are just checking different skills, or even tricking the robots into looking safe when they aren't?

This paper is like a detective story where the authors go undercover to audit four of these popular safety tests. They didn't just take the scores at face value; they ran the tests themselves on up to 22 different AI models from various tech companies to see what was really happening.

First, they discovered a sneaky loophole in one of the most famous tests, called R-Judge. This test uses a scoring system called "F1" to grade how well a model spots dangerous requests. The authors found a weird trick: if a robot just guesses "DANGER!" for every single question, it actually gets a pretty high score—0.690. In fact, this consistent, "always-unsafe" robot scored higher than five real, smart models that were actually trying to distinguish between good and bad requests! It's like a teacher giving a student a high grade just for raising their hand and shouting "Fire!" every time, even if there is no fire. The paper proves that this specific scoring method is broken because it doesn't give credit for correctly identifying safe things.

Next, the team looked at whether the robots' general intelligence (their "capability") was just hiding behind the safety scores. They found that being smart does help a robot pass some safety tests, but it's a messy relationship. Sometimes, the smarter the robot, the safer it seems. Other times, being super smart actually made the robot less safe in specific scenarios, like when it was asked to pretend to be a villain in a story. The authors showed that you can't just assume a smart robot is a safe robot; the connection changes depending on which test you use and which group of robots you are looking at.

The most surprising finding was that these safety tests don't agree with each other. The authors took 18 robots and ran them through three different major safety tests. The results were chaotic: a robot that was ranked as the "safest" on one test could be ranked near the "least safe" on another. It's like a student getting the top score in Math, but the bottom score in Science, and then the test organizers arguing about who is actually the "best student." The paper found that if you only look at a tiny group of robots (like just seven), you might see a fake pattern that looks like a trade-off (where being safe in one way means being unsafe in another). But when they expanded the group to 18 or more robots, that pattern disappeared. The "trade-off" was just a fluke caused by looking at too few examples.

Finally, the authors checked if these safety scores could predict how a robot would behave in a totally new situation it hadn't seen before. They found that the robots' general intelligence was great at predicting if they could finish a task (like buying a ticket online), but the safety scores were terrible at predicting if the robot would do something dangerous in a new scenario. The only test that showed a strong link to a specific type of danger was one called AgentHarm, which predicted how well a robot resisted "jailbreaks" (tricks to make it break its rules). However, even this wasn't a perfect measure of general safety; it just meant the robot was good at resisting that specific type of trick.

In short, the paper concludes that there is no single "safety score" for AI. You cannot just look at one number and say, "This robot is safe." The score you get depends entirely on which test you use, how you calculate the grade, and which robots you are testing. If you want to make a claim about a robot's safety, you have to be very specific: name the exact test, the exact metric, the behavior you are measuring, and the group of robots you tested. Otherwise, you might just be looking at a robot that is really good at guessing "Danger!" or one that is just lucky with a small group of friends.

Technical Summary: Safety, or Just Capability? A Validity Audit of Agent-Safety Benchmarks

Problem Statement
The rapid proliferation of benchmarks claiming to measure the safety of Large Language Model (LLM) agents has led to a critical validity crisis. Recent taxonomies indicate that safety rankings are highly inconsistent; swapping benchmarks often results in contradictory model rankings with near-zero concordance. The core issue is that safety scores are frequently treated as interchangeable measures of a single property ("safety"), yet they may measure distinct constructs, be confounded by general model capability, or rely on invalid metrics that reward degenerate behaviors. This paper treats four prominent benchmarks—R-Judge, InjecAgent, AgentHarm, and AgentDojo—as measurements requiring validation rather than settled truths.

Methodology
The authors conducted a comprehensive validity audit involving up to 22 models from nine organizations (OpenAI, Meta, Qwen, Mistral, DeepSeek, Amazon, Anthropic, Google, Cohere). The study employed a three-pronged validation framework:

Metric Validity: Analyzing whether the headline metrics (e.g., F1) validly measure the intended target behaviors or allow for high-scoring degenerate policies.
Construct Validity: Determining if distinct benchmarks measure a common underlying factor or dissociable ones, and quantifying the extent to which general capability (measured via MMLU and GPQA-Diamond) confounds safety comparisons.
Criterion Validity: Testing whether safety scores predict held-out behavioral outcomes (task success, agentic misalignment, and jailbreak safety) beyond what general capability predicts.

The analysis utilized Spearman correlations, partial correlations controlling for a capability composite, and bootstrap resampling (including organization-clustered and leave-one-out analyses) to assess stability. The study pre-specified hypotheses and decision rules to avoid p-hacking, though it notes these were pre-specified rather than preregistered.

Key Contributions and Results

Metric Validity Failure (RQ4): The paper identifies a formal failure in the R-Judge benchmark's use of the F1 score. Because R-Judge has a near-balanced base rate of "unsafe" items ( $\pi \approx 0.527$ ), a degenerate policy that labels every trace "unsafe" achieves an F1 score of 0.690. This constant-baseline score outranks five of the 21 evaluated models that actually discriminate between safe and unsafe traces. The authors argue that F1 is blind to true negatives in this context and cannot stand alone as a valid measure of two-sided discrimination.
The Capability Confound (RQ2): Capability is a significant confound, but its effect is metric-dependent. While capability correlates strongly with R-Judge's balanced accuracy and F1, it correlates weakly with specificity. Furthermore, the relationship between capability and safety metrics shifts depending on the model panel size; for instance, the correlation between R-Judge specificity and AgentHarm safety was $-0.64$ on a small 7-model panel but dissolved to $+0.02$ on the full 18-model panel.
Benchmark Disagreement and Small-Panel Artifacts (RQ1): The four benchmarks rank the same 18 models differently. A perceived trade-off between R-Judge specificity and AgentHarm safety, observed in small subsets, disappears in the full panel. The authors demonstrate that small panels (e.g., $n=7$ ) can produce spurious correlations (where $|\rho| \ge 0.5$ in 25% of random subsets) that do not persist as the model population expands.
Criterion Validity Variance (RQ3): The predictive validity of safety scores depends entirely on the chosen outcome:
- Task Success: Capability predicts task success ( $\rho = +0.60$ ), but safety scores add no incremental predictive value once capability is controlled.
- Misalignment Safety: Capability correlates negatively with misalignment safety ( $\rho = -0.44$ ) on the initial panel, suggesting higher capability models may be more prone to specific misalignments (e.g., blackmail/leaking). However, this correlation weakens to $-0.16$ on an expanded 41-model panel and is not statistically significant.
- Jailbreak Safety: AgentHarm shows the strongest held-out association with jailbreak safety ( $\rho = +0.72$ after controlling for capability). However, this is largely evidence of convergent validity, as both instruments measure harmful compliance. The claim that AgentHarm is selectively better at predicting jailbreak over misalignment rests on borderline evidence ( $p=0.051$ under organization resampling).

Significance and Claims
The paper concludes that a capability score is not a safety score, and no single agent-safety benchmark can stand in for safety as a whole. The validity of a safety claim is contingent on the specific benchmark, metric, target behavior, and model population used to produce it.

The authors emphasize that small panels are a critical threat to validity, where weak relationships can appear systematic. They argue that safety evaluations must report the full confusion structure, avoid standalone F1 scores where true negatives matter, and explicitly tie every claim to a specific target behavior and model population. The study does not claim to have solved the problem of agent safety but rather provides a rigorous audit demonstrating that current safety rankings are not interchangeable and require careful, context-specific interpretation.

Technical Summary: Safety, or Just Capability? A Validity Audit of Agent-Safety Benchmarks

More like this