🤖 AI

Deployment-Relevant Alignment Cannot Be Inferred from Model-Level Evaluation Alone

This paper argues that deployment-relevant alignment cannot be inferred from model-level evaluations alone, demonstrating through audits and stress tests that current benchmarks lack user-facing verification and process steerability, thereby necessitating a shift toward system-level evaluation frameworks that explicitly account for interactional contexts and scaffold dependencies.

Original authors: Varad Vishwarupe, Nigel Shadbolt, Marina Jirotka, Ivan Flechais

Published 2026-05-07

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Varad Vishwarupe, Nigel Shadbolt, Marina Jirotka, Ivan Flechais

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Idea: The "Car Test" vs. The "Road Trip"

Imagine you want to know if a new self-driving car is safe to drive on real highways.

Current Practice (Model-Level Evaluation):
Right now, researchers test the car's "brain" (the AI model) in a quiet, empty garage. They ask it: "If I press the brake, does it stop?" or "If I ask for the time, do you give the right time?"
If the car answers correctly in the garage, we assume it's safe to drive anywhere. We give it a high score and say, "This car is aligned with safety rules."

The Paper's Argument:
The authors say this is a dangerous mistake. Just because the car's brain works perfectly in a garage doesn't mean it will handle a rainy highway, a confused passenger, or a sudden detour.

The Garage = The current benchmarks (testing the model alone).
The Highway = The real world (where the AI is actually used).

The paper argues that you cannot infer how the car behaves on the highway just by testing it in the garage. To know if the AI is truly "aligned" (safe and helpful) in the real world, you have to test it while it's driving, with passengers, and with traffic.

The Four Levels of Testing

The authors break down AI testing into four different "floors" of a building. Current testing mostly happens on the second floor, but we are making claims about the fourth.

Floor 1: The Model (The Brain): Testing the raw code and weights. ("Does the math work?")
Floor 2: The Response (The Answer): Giving the model a fixed question and grading its single answer. ("Did it answer '42' when asked 6x9?") This is where almost all current testing happens.
Floor 3: The Interaction (The Conversation): Watching how the AI talks over several turns. Does it ask for clarification? Does it let the user change the plan? Does it admit when it's unsure?
Floor 4: The Deployment (The Real Job): The AI working inside a specific company, with real rules, real bosses, and real users.

The Problem: We are testing on Floor 2, but we are making promises about Floor 4. The gap between these floors is huge.

The Two Big Studies (The Evidence)

The authors didn't just say this; they did two studies to prove it.

Study 1: The "Toolbox" Audit

They looked at 11 of the most famous AI testing tools (benchmarks) to see what they actually measure. They used a checklist of 8 important "interaction skills," such as:

Verification Support: Does the AI show its work so the user can check if it's right? (Like showing your math homework).
Process Steerability: Can the user tell the AI to change how it solves the problem? (Like saying, "No, try a different route.")

The Findings:

The "Verification" Vacuum: Zero out of the 11 benchmarks checked if the AI helps the user verify the answer. They only checked if the answer was "correct," not if the user could trust it.
The "Steerability" Gap: Almost no benchmark checked if the user could control the process.
Fragmentation: The few tests that did check interaction skills were scattered. One tested memory, another tested tool use, but none tested the whole picture. It's like having a toolbox where one tool is a hammer, another is a screwdriver, but you have no wrench.

The Metaphor: Imagine judging a chef only by how well they chop onions in a silent kitchen. You never see them taste the soup, ask the customer if they are allergic to nuts, or adjust the recipe based on the customer's mood. You don't know if they are a good chef, only a good onion chopper.

Study 2: The "Same Car, Different Roads" Stress Test

The authors took three different AI models (Claude, GPT-4o, and Llama) and kept their "brains" exactly the same. Then, they changed the scaffolding (the instructions and interface wrapping the model).

They gave the models four different "outfits":

Plain Chat: Just talk.
Clarification Mode: Force the AI to ask questions before answering.
Plan Mode: Force the AI to show a plan before acting.
Verification Mode: Force the AI to show its assumptions and how to check the answer.

The Shocking Result:

Model A (Claude): When put in the "Verification Mode," it became amazing at helping users check answers.
Model B (GPT-4o): When put in the exact same "Verification Mode," it didn't change at all. It stayed the same.
Model C (Llama): It barely changed, and in some cases, it got worse at following instructions.

The Lesson: The "outfit" (the system design) matters just as much as the "brain" (the model). You cannot predict how a model will behave in a real system just by looking at the model alone. The same instructions work wonders for one AI and do nothing for another.

What Should We Do Instead?

The paper proposes a new way to report AI results, called an "Alignment Profile."

Instead of saying: "This AI has a score of 95/100 and is safe for hospitals." (This is a lie based on garage testing).

We should say: "This AI, when used with [Specific Instructions] and [Specific User Interface], showed these specific behaviors. We did not test it in a real hospital, so we cannot claim it is safe for that specific job."

The Proposal:

Stop pretending one score covers everything.
Test the whole system: Test the model plus the instructions plus the user interface together.
Be honest about the gap: Clearly state, "We tested this in a garage. We haven't tested it on the highway yet."

Summary

The paper argues that AI alignment is not a property of the brain alone; it is a property of the whole system. You can't judge a self-driving car by testing its engine in a lab. You have to drive it on the road, with a passenger, in the rain. Until we start testing AI in that context, our claims about its safety and helpfulness are just guesses.

Technical Summary: Deployment-Relevant Alignment Cannot Be Inferred from Model-Level Evaluation Alone

Problem Statement

Current alignment evaluation in machine learning predominantly focuses on model-level or response-level metrics. Influential benchmarks score model outputs under fixed inputs (e.g., truthfulness, instruction compliance, pairwise preference) and treat these scores as sufficient evidence for claims about deployment alignment.

The paper argues that this inference is invalid. Deployment-relevant alignment is a system-level phenomenon dependent on the interaction between the model ( $M$ ), the scaffolding ( $S$ —prompts, memory, tools, UI), and the deployment context ( $C$ —user population, oversight, policy). Because current benchmarks hold $S$ and $C$ fixed or ignore them, they cannot measure properties essential to deployed systems, such as verification support, process steerability, or the distribution of repair burden. The paper posits a "non-identifiability" problem: evaluating $M$ alone cannot distinguish between different deployments of the same model that exhibit vastly different alignment-relevant behaviors due to differences in $S$ .

Methodology

The paper employs two primary empirical studies to support its position:

1. Structured Benchmark Audit

Scope: A dual-coded audit of 11 primary benchmarks (plus a meta-evaluation control, JudgeBench) and an extension to 16 benchmarks total.
Rubric: An eight-dimension rubric derived from interactional and deployment-level alignment literature (Terry et al., Shen et al., and empirical work on grounding/repair). The dimensions are:
- D1: Specification Alignment (handling under-specified requests).
- D2: Process Alignment/Steerability (user control over the reasoning process).
- D3: Evaluation Alignment/Verification Support (helping users verify outputs).
- D4: Multi-turn Context Retention.
- D5: Grounding/Common Ground.
- D6: Repair Burden Distribution.
- D7: Personalization/User-Specific Alignment.
- D8: Workflow/Oversight Realism.
Protocol: Two trained external coders independently scored benchmarks on a 0–2 anchored scale (0: absent, 1: partial/incidental, 2: substantive). Disagreements were resolved via anchor-driven reconciliation.
Reliability: Cohen's $\kappa$ was 0.87 across the 128 cells of the 16-benchmark corpus.

2. Blinded Cross-Model Stress Test

Design: A study holding model weights ( $M$ ) and deployment context ( $C$ ) approximately fixed while varying only the interaction scaffolding ( $S$ ).
Variables:
- Models: Three frontier models (Claude-Opus-4.7, GPT-4o, Llama-4-Scout).
- Scaffolds: Four conditions ranging from a plain chatbot (C1) to explicit verification-support prompts (C4).
- Data: 180 transcripts (60 per model) across medical, legal, and technical domains.
Protocol: Transcripts were blinded, stripped of system prompts, and coded by two raters for D1, D2, and D3. Disagreements were resolved using a conservative-min policy.

Key Findings

From the Benchmark Audit

The D3 Vacuum: Verification support (D3) is absent across all 11 audited benchmarks. No benchmark explicitly evaluates whether a model's output helps a user verify, critique, or understand the answer.
Near-Absence of Steerability: Process steerability (D2) is nearly absent, scoring non-zero in only one benchmark ( $\tau$ -bench) and even there only as a partial score, not as explicit user control.
Fragmentation of Interactional Benchmarks: The four identified interactional benchmarks (CURATe, Rifts, $\tau$ -bench, Common Ground) measure different, non-overlapping subsets of dimensions. There is no unified evaluation target for interactional alignment.
Construction Determines Measurement: The structure of the evaluation (how the benchmark is constructed) determines what is measured, not the data source. For example, WildBench uses real user logs but scores single-response checklists, failing to capture interactional properties despite its ecological realism.

From the Cross-Model Stress Test

Model-Dependent Scaffold Efficacy: The same scaffolding ( $S$ $S$ ) produces categorically different behavioral outcomes across models.
- Claude-Opus-4.7: Highly plastic; the C4 verification scaffold lifted its D3 score from 0.42 to 2.00 (ceiling).
- GPT-4o: Saturated; the same C4 scaffold had zero measurable effect on D3 (remained at 0.33).
- Llama-4-Scout: Partially inverted; showed weak sensitivity and categorical absence of D2 across all scaffolds.
Implication: The gap identified in the audit (missing measurements of interactional properties) cannot be closed at the model level because scaffold efficacy is itself a function of the model weights.

Key Contributions

Level-Indexed Framing: The paper proposes distinguishing four levels of alignment evaluation: Model-level (weights/objectives), Response-level (outputs under fixed inputs), Interaction-level (multi-turn properties), and Deployment-level (system-in-workflow). It argues claims must be indexed to the level of evidence.
Structured Audit Evidence: Provides empirical evidence that the current benchmark ecosystem systematically fails to measure critical interactional dimensions (specifically D2 and D3).
Non-Identifiability Proof: Demonstrates via the stress test that $B = f(M, S, C)$ cannot be inferred from $M$ alone, as $S$ interacts with $M$ in model-dependent ways.
Evaluation Agenda: Proposes a shift from single alignment scores to alignment profiles. This includes:
- Reporting evidence across the eight dimensions.
- Using fixed-scaffolding protocols for comparable interactional evaluation.
- A six-row reporting template (Appendix G.1) that forces authors to explicitly state the scope of their claims (which $M, S, C$ configurations the evidence supports).

Significance and Claims

The paper claims that the field's reliance on response-level benchmarks to license deployment-aligned claims is structurally flawed. The "inferential gap" between benchmark performance and deployed behavior is larger than currently acknowledged.

The significance of the work lies in:

Correcting the Scope of Claims: Arguing that model-level evidence is insufficient for deployment-level claims and that adding more response-level benchmarks will not fix the problem.
Methodological Discipline: Introducing a framework where alignment claims are explicitly scoped to the level of evidence collected, preventing the conflation of model properties with system behaviors.
Empirical Validation: Providing direct evidence that interactional properties are not intrinsic to the model but are emergent properties of the system ( $M+S+C$ ), and that these properties vary significantly across models even with identical scaffolding.

The paper does not claim that model-level evaluation is useless, but rather that it is insufficient for deployment claims. It calls for the community to develop evaluative vocabularies and protocols that account for the system, the interaction, and the user, rather than focusing solely on the model.