Technical Summary: Validating Comparative LLM Safety Scoring Without Ground-Truth Labels

1. Problem Formulation

The paper addresses a critical gap in the deployment of Large Language Models (LLMs): the need to compare candidate models for safety in specific languages, sectors, or regulatory regimes where no ground-truth annotated safety benchmark exists.

In many deployment scenarios (e.g., a Norwegian public-sector agency piloting a local model), teams must decide which model is safer under specific constraints without access to labeled datasets. Existing paradigms fail to fully address this:

• Static Benchmarks: Require ground-truth labels, are often English-centric, and freeze evaluation at release time.
• Discovery-Oriented Auditing: (e.g., automated red-teaming) generates transcripts and hypotheses but lacks a committed, repeatable scoring mechanism with uncertainty quantification suitable for procurement.
• LLM-as-Judge: While pairwise comparisons are more reliable than absolute scores, they inherit biases and instability. Without a validation chain, scores cannot be defended as evidence of safety differences rather than artifacts of the auditing apparatus.

The authors define this space as benchmarkless comparative safety scoring. The core challenge is to produce defensible comparative evidence (scores, deltas, critical rates) without ground-truth labels, ensuring the instrument responds to target behavior rather than noise or auditor/judge artifacts.

2. Methodology: The Instrumental-Validity Chain

To validate a scoring instrument in the absence of ground truth, the authors propose an instrumental-validity chain. This chain replaces ground-truth agreement with three specific requirements:

1. Responsiveness: The tool must distinguish between a known safety-relevant contrast. The authors use a safe-vs-abliterated contrast (comparing a safety-aligned model against a capability-matched variant where refusal behavior has been ablated via the refusal direction).
2. Target Sensitivity (Variance Dominance): The variance in scores must be driven primarily by the target model, not by the auditor (prober) or judge (grader) artifacts. This is quantified via variance decomposition ($\eta^2$).
3. Reproducibility: Scores must stabilize across multiple reruns ($n$) to ensure that deltas between targets are not obscured by stochastic noise.

The SimpleAudit Implementation

The authors instantiate this chain in SimpleAudit, a local-first Python library and measurement instrument.

• Architecture: It separates three independent roles: Target (T), Auditor (A), and Judge (J).
• Workflow: A scenario pack initializes a conversation; the Auditor generates probes; the Target responds; the Judge grades the transcript with a structured verdict (severity 0–4).
• Scoring: Severity labels are remapped to a 0–100 scale. The instrument reports aggregate scores, matched deltas ($\Delta$), critical rates (frequency of severe failures), and uncertainty intervals.
• Design Commitments: Local execution (no external API dependency for sensitive data), role modularity, explicit configuration, and versioned scenario packs.

3. Key Contributions

The paper makes six primary contributions:

1. Category Definition: Identifies "benchmarkless comparative safety scoring" as a distinct evaluation category with its own claim contract.
2. Validation Framework: Specifies the instrumental-validity chain (responsiveness, target sensitivity, reproducibility) required to validate such tools.
3. Reference Implementation: Introduces SimpleAudit, a tool that packages fixed scenarios, rubrics, and roles into a repeatable instrument.
4. Empirical Validation: Validates SimpleAudit on a Norwegian safety pack, demonstrating it meets all three chain requirements.
5. Generalization Check: Applies the same validation chain to Petri (a discovery-oriented auditing tool), showing that the chain admits Petri as well, provided specific dimensions are selected.
6. Deployment Evidence: Demonstrates the application of this framework in a Norwegian public-sector procurement case comparing Borealis and Gemma 3.

4. Key Results

Validation of SimpleAudit

• Responsiveness: SimpleAudit successfully separates safe and abliterated targets. Across reliable judge-auditor combinations, AUROC values range from 0.89 to 1.00.
• Target Sensitivity: Variance decomposition shows the target model is the dominant factor ($\eta^2 \approx 0.52$), while auditor and judge contributions are substantial but secondary. This confirms scores reflect target behavior rather than apparatus artifacts.
• Reproducibility: Scores stabilize rapidly. By 10 reruns, the mean absolute deviation from the reference score drops to roughly 1 point on a 0–100 scale, well below the deltas driving procurement decisions.
• Configuration Insights:
  • Auditor Selection: The auditor is the most critical design point. An auditor that is too strong (e.g., frontier models) compresses safe-target scores, erasing the comparative deltas the instrument is meant to report. The auditor must match the target's capability range.
  • Judge Selection: While absolute scores vary by judge, pairwise deltas remain stable. Judges with a critical-miss rate near 10% (relative to a frontier reference) are viable for governance.
  • Cost Efficiency: Setting the Judge capability equal to the Auditor ($J=A$) is the recommended default, saving tokens without sacrificing comparative signal.

Comparison with Petri

• The validation chain admits Petri as well. Specifically, the "concerning" dimension in Petri passes all three requirements (AUROC $\ge$ 0.99, target dominance, stability).
• However, the tools differ in their upstream claim contracts. SimpleAudit enforces a single, committed scenario pack and aggregation rule, making it operationally fit for procurement. Petri offers flexibility for discovery but requires the user to commit to constructs and aggregation rules to generate comparable scores.

Norwegian Procurement Case Study

• Comparing Borealis and Gemma 3 across sizes (1B–27B) revealed that "safer" is not a monolithic property.
• Borealis generally showed higher scores and lower critical rates, but the advantage was category-dependent (e.g., strong in Healthcare and Public Sector, weaker in Language).
• Scaling: Safety improvements plateaued at 12B for some categories, contradicting a simple "larger is safer" rule.
• Conclusion: The evidence supports a nuanced view: the safer model depends on the scenario category and the specific risk measure (mean score vs. critical rate).

5. Significance and Claims

The paper claims that benchmarkless comparative safety scoring is a distinct evaluation category, not a degraded form of benchmarking. Its significance lies in:

• Operationalizing Safety for Regulated Domains: It provides a method for teams in regulated environments (e.g., public sector, non-English languages) to generate defensible comparative evidence without waiting for labeled benchmarks.
• Shift from Ranking to Evidence: The authors argue against collapsing results into a single leaderboard. Instead, valid deployment evidence consists of a bundle: scores, matched deltas, critical rates, uncertainty intervals, and sensitivity checks, all tied to a specific instrument configuration.
• Falsifiability: By explicitly stating the "claim contract" (fixed instrument, scenario pack, roles), the evaluation becomes falsifiable. Other groups can rerun the same instrument or swap components to challenge the results.
• Local-First Governance: The design ensures that prompts, transcripts, and policies do not need to leave the deployment environment, addressing data sovereignty and privacy concerns in regulated sectors.

The authors are modest about their claims: passing the instrumental-validity chain is necessary but not sufficient for deployment. It proves the instrument works as a measurement tool, but domain experts must still judge whether the scenario pack captures the actual deployment construct. The contribution is the category and the validation chain, with SimpleAudit serving as a stopgap implementation for the procurement niche.