Technical Summary: How Should AI Safety Benchmarks Benchmark Safety?

Problem Statement

AI safety benchmarks are currently the primary mechanism for evaluating and mitigating risks in advanced AI systems. However, this paper argues that the current state of AI safety benchmarking suffers from significant technical, epistemic, and sociotechnical shortcomings. The authors identify that the field largely overlooks established safety engineering principles, risk management frameworks, and measurement theory developed over decades. Consequently, existing benchmarks often fail to provide a valid, reliable, or actionable basis for asserting the safety of AI systems in real-world deployment. The core problem is a misalignment between the abstract, capability-focused nature of traditional benchmarking and the normative, risk-mitigation goals required for AI safety.

Methodology

To diagnose these limitations and propose improvements, the authors conducted a comprehensive scoping review of 210 AI safety benchmarks. The study employed a three-dimensional analytical framework grounded in established scientific disciplines:

1. Construct Coverage (Risk Management): The authors applied the Rumsfeld matrix (categorizing risks as known knowns, known unknowns, unknown knowns, and unknown unknowns) to map the risks evaluated by current benchmarks. This revealed systematic imbalances in which risks are monitored versus those that are neglected.
2. Risk Quantification (Probabilistic Risk Assessment): Drawing from safety engineering, the study analyzed how benchmarks quantify risk. It examined whether benchmarks treat empirical frequencies as calibrated probabilities and how they operationalize the severity of consequences (e.g., using binary pass/fail rates vs. severity-weighted metrics).
3. Measurement Validity (Measurement Theory): The authors evaluated benchmarks against principles of measurement science, specifically focusing on standardization, accuracy, precision, and construct validity. They investigated whether benchmark scores serve as defensible proxies for real-world harms or if they suffer from "proxy chains" that erode validity.

The review involved iterative manual coding of the 210 benchmarks until thematic saturation was reached, ensuring the identified patterns were systemic rather than artifacts of the sample.

Key Findings

The analysis of 210 benchmarks revealed three core limitations:

1. Imbalanced Construct Coverage: 81% of surveyed benchmarks evaluate only "known knowns" (predefined, empirically verified risks like toxicity or fixed jailbreaks). "Known unknowns" (anticipated but not fully understood emergent behaviors) receive limited attention (16%), while "unknown knowns" (theoretically understood risks like distribution shift) and "unknown unknowns" (entirely unforeseen failures) are nearly absent (2% and 1%, respectively). This creates structural blind spots where systems optimized for known risks remain vulnerable to novel failure modes.
2. Lack of Probabilistic Rigor: 79% of benchmarks reduce safety assessment to binary pass/fail rates, treating empirical frequencies as calibrated probabilities without accounting for uncertainty or severity. When severity scales are used, they often lack principled justification, relying on ad hoc author judgments rather than established frameworks.
3. Erosion of Measurement Validity: Metrics often fail to link to real-world outcomes. For instance, a reduction in a toxicity score does not necessarily correlate with a proportional reduction in actual harm. Benchmarks frequently rely on static, single-turn interactions that fail to capture the sociotechnical context where safety emerges, leading to a disconnect between benchmark scores and deployment risks.

Recommendations

Based on these findings, the authors propose ten recommendations (R1–R10) organized into three categories to improve the validity and usefulness of AI safety benchmarks:

I. Expanding Construct Coverage

• R1: Document known blind spots explicitly, specifying which risks are excluded.
• R2: Expand boundaries by investing in open-ended evaluation methods (e.g., fuzzing, red-teaming, multi-agent stress testing) to discover novel risks.
• R3: Reframe well-understood ML phenomena (e.g., distribution shift, data contamination) as safety-critical issues rather than mere performance limitations.

II. Adopting Principled Risk Quantification

• R4: Calibrate benchmark frequencies to real-world exposure estimates (e.g., weighting refusal rates by in-the-wild query prevalence) rather than reporting raw frequencies as probabilities.
• R5: Ground severity scales in principled frameworks (e.g., regulatory standards, domain-specific ethics) rather than arbitrary ordinal scales.
• R6: Account for uncertainty quantification by reporting confidence intervals, inter-rater reliability, and applying safety margins when extrapolating to deployment.

III. Aligning with Measurement Theory

• R7: Standardize safety constructs with transparency, defining operational definitions and acknowledging contested normative choices.
• R8: Ensure reproducibility through locking and versioning of models, prompts, and evaluation parameters.
• R9: Anchor proxies in deployment contexts by using real-world user data (e.g., WildChat) and validating synthetic data against actual usage patterns.
• R10: Implement iterative refinement via community input, involving affected communities to assess whether scenarios reflect actual harms and to capture value disagreements.

Illustrative Application

To demonstrate the feasibility of these recommendations, the authors developed a proof-of-concept benchmark evaluating Large Language Models (LLMs) in U.S. adolescent mental-health scenarios. This benchmark:

• Used prevalence data (YRBS) to weight risk categories.
• Grounded severity in the CANS-MH action levels.
• Evaluated models across multi-turn interactions.
• Calculated "exposure-weighted potential harm" by combining benchmark failure rates with platform reach and user prevalence.
• Result: The study showed that models with similar raw refusal rates could have vastly different estimated real-world burdens (e.g., one model had a 14x higher estimated burden than another in suicide-attempt scenarios), illustrating the necessity of prevalence calibration.

Significance and Claims

The paper claims to advance the science of benchmarking by providing a roadmap to transform AI safety evaluation from a capability-focused exercise into a rigorous, normative process that connects abstract values to real-world outcomes.

• For Researchers: It offers a framework to design benchmarks that are epistemologically sound, covering a broader spectrum of risks and utilizing probabilistic rigor.
• For Practitioners: It provides a checklist and workflow-oriented questions to help deploy AI systems more responsibly by understanding the limitations of current scores and the gap between benchmark performance and deployment safety.
• For the Field: The study argues that adhering to established risk management principles and measurement theory is essential to prevent false confidence in AI safety. It posits that without these improvements, benchmarks risk fostering a "false sense of security" that distracts from systemic risks and perpetuates biases.

The authors conclude that while current benchmarks are useful for identifying specific vulnerabilities, they are insufficient for asserting deployment safety. A shift toward system-level assessment, probabilistic quantification, and participatory design is necessary to build truly safe AI systems.