🤖 AI

When benchmark inferences do not compose: Projectibility in AI evaluation

This paper argues that valid AI benchmark inferences do not automatically compose into warranted consequential claims due to misalignments in endpoints, assumptions, and dependencies, proposing a "projectibility audit" framework to diagnose these unsupported logical joins.

Original authors: Brett Reynolds

Published 2026-07-30

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Brett Reynolds

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out if a new robot chef is ready to run a busy restaurant. You can't just watch it chop onions in a test kitchen and assume it will handle a full dinner service. You have to ask: Will it cook the same way when the ingredients are different? Will it still be good if a human manager checks the food? Will it survive a week of service, or just one hour? This is the heart of AI evaluation, a field where scientists try to measure how smart or capable artificial intelligence really is.

To do this, researchers often use benchmarks. Think of a benchmark as a standardized test, like a driver's ed exam. If a car passes the test, it gets a license. But here's the tricky part: passing a test doesn't automatically mean the car can drive safely in a snowstorm, on a bumpy dirt road, or with a nervous passenger. In the world of AI, we often see a chain of reasoning: "The AI passed the test, so it has 'general reasoning' skills, so it can do this specific job, so it's safe to use." The big question is: Does the evidence from step one actually connect to step two, and does step two connect to step three? This paper explores why that connection often breaks, even when every single step looks perfect on its own.

The Broken Chain of Logic

Imagine you are building a tower out of blocks. You have a block that is perfectly square and sturdy (a benchmark result). You have another block that is also perfectly square and sturdy (a local study showing the AI works in a specific office). You might think, "If I stack them, I have a tall, stable tower!" But what if the top of the first block is slightly rounded, and the bottom of the second block is slightly flat? They look like they fit, but they don't actually touch. The tower wobbles and falls.

This is the main problem Brett Reynolds is pointing out in this paper. He calls it the non-composition principle. In simple terms, just because you have two pieces of evidence that are both "warranted" (supported by good data) doesn't mean you can glue them together to make a bigger, stronger claim. The paper argues that when we try to chain these pieces of evidence together—like going from "AI passed a test" to "AI is safe for real-world use"—we often miss the tiny gaps where the logic falls apart.

The "Legal Assistant" Puzzle

To show how this happens, the author creates a story about a law firm trying to use a new AI tool.

The Test: A developer shows that their AI model gets 90% of legal questions right on a specific test.
The Human Check: The law firm shows that when their human lawyers review drafts, they catch 99% of the mistakes.
The Big Leap: The firm assumes that if they combine these two, the final legal memos will be 99.9% perfect.

The paper says: Stop right there. This logic is broken.

Why? Because the "Test" and the "Human Check" are talking about different things. The test might only check if the AI can find a quote from a book. But the human lawyers might be checking if the AI missed a crucial law that wasn't in the book. If the AI misses that law, the human lawyer might not catch it because they are only looking for the wrong kind of mistake. The two studies are like two people looking at a puzzle from different angles; they both see a perfect picture, but they aren't looking at the same picture.

The paper simulates this with numbers. In one scenario, the AI does great on "Termination Clause" requests (a specific type of legal question) but fails miserably on "Reasonable Notice" requests. If you just average the scores, you might think the AI is great overall. But if the law firm needs to use it for both types of questions, the "average" score is lying to them. The AI is actually a disaster for half the job.

The "Magic Mean" Trap

The paper also warns about a statistical trick called aggregation. Imagine you have a class of students. Half of them get 100% on a math test, and the other half get 0%. The average score is 50%. That sounds okay, right? But if you need every student to pass to run a school bus, that average is useless.

In AI, a "stable mean" (a steady average score) can hide a lot of trouble. The author shows that an AI's score might stay the same on average, but the types of mistakes it makes could be getting much worse. Maybe it used to make small typos, but now it makes dangerous legal errors. If you only look at the average, you miss the fact that the "worst-case" scenarios are getting much worse. The paper uses a simulation to show that you can have a perfectly stable average score while the AI is actually failing in the most critical ways.

What This Means for the Future

So, what does the paper suggest we do? It proposes a "Projectibility Audit."

Think of this as a checklist for anyone trying to use AI. Before you say, "This AI is ready for the real world," you have to answer specific questions:

Do the blocks fit? Does the "Test" actually cover the same types of questions as the "Real World" job?
Are the conditions the same? Did the test happen in a quiet room, while the real job is in a noisy, chaotic office?
Did we lose information? Did we throw away the details about which questions were hard, so we only see the average?

The paper doesn't say AI is bad or that we can't use it. It says we need to be much more careful about how we connect the dots. We can't just assume that because an AI passed a test, it will pass a job. We have to check the connection between the test and the job, and the job and the final result.

The Bottom Line

This paper is a reality check for the AI world. It suggests that we are often too quick to stack our evidence blocks, assuming they fit perfectly when they actually have gaps. The author uses simulations and logical arguments to show that support for one step does not automatically transfer to the next step.

If a developer says, "Our AI is 90% accurate," and a company says, "Our lawyers catch 99% of errors," the paper tells us: Don't multiply those numbers yet. You have to check if the AI is making the same kind of errors the lawyers are looking for. If you don't, you might build a tower that looks tall but collapses the moment you put a real load on it. The paper doesn't claim to have solved the problem of AI safety, but it gives us a better map to find where the cracks in our logic are hiding.

Technical Summary: When Benchmark Inferences Do Not Compose

1. The Problem: Non-Composition of Warranted Links

The paper addresses a critical epistemic gap in AI evaluation: the assumption that individually warranted inference steps automatically compose into a warranted chain. In standard AI evaluation workflows, a benchmark result (e.g., a high score on a multidomain battery) is often treated as evidence for a consequential claim (e.g., authorization for professional use). This chain typically involves multiple steps:

Generalization: Extending benchmark performance to new items under the same rules.
Interpretation: Attributing the score to a specific capability (e.g., "legal reasoning").
Extrapolation: Projecting that capability to a new task or system configuration (e.g., a retrieval-augmented application).
Transport: Moving the claim to a different population, site, or time.
Combination: Integrating the system's output with human review and downstream policy consequences.

The central problem identified is that validity-centred approaches, which require evidence for each claim, often fail to account for the interfaces between these claims. Even if a benchmark study is sound and a separate deployment study is sound, their combination may not warrant the final claim. The paper argues that "warranted links don't automatically make a warranted chain."

This failure arises from three primary sources:

Endpoint Mismatch: The target of one study (e.g., a base model answering supplied-text questions) is not the source of the next (e.g., a deployed application with retrieval and human review).
Hidden Dependencies: Shared data, model lineages, or assumptions can make apparently independent studies dependent, eroding the cumulative strength of the evidence.
Information Loss: Aggregated metrics (e.g., a stable mean score) can erase item-level distinctions (e.g., concentrated failures or specific error types) that are critical for downstream projections.

2. Methodology: The Projectibility Audit

The paper proposes a projectibility audit as a diagnostic tool within an argument-based validity framework (drawing on Kane, 2013; Messick, 1995). Unlike frameworks that focus solely on construct meaning or causal transportability, this approach specifically targets the interfaces between adjacent inference steps.

Core Concepts

Projectibility: Defined as whether a bounded extension from observed to unobserved cases is warranted. It is not a scalar property of a system but a claim-specific status (supported, defeated, or unresolved).
Empirical Nodes: The audit represents each study or inference step as a typed node $N = \langle \text{object, population, conditions, outcome, period} \rangle$ .
Projection Edges: An edge $e$ connects a source node to a target node, defined by a claim, assumptions, and evidence.

The Non-Composition Principle

The paper formalizes the core failure mode as:
$\text{Warr}(C_1) \land \text{Warr}(C_2) \not\Rightarrow \text{Warr}(C_2 \circ C_1)$
Where $C_1$ and $C_2$ are adjacent links. Composition is only warranted if:

Endpoint Alignment: The target of $C_1$ matches the source of $C_2$ across five fields: object, population, conditions, outcome, and period.
Warrant Transmission: Assumptions and effect modifiers are compatible, and dependence/uncertainty are propagated rather than reset.

The Audit Procedure

The methodology involves a ten-step declaration process to be completed before data analysis:

Define Source-Target Claims: Explicitly state the observed data and the new claim.
Type Endpoints: Record the five fields for source and target.
Name the Extension: Specify the type of projection (generalization, extrapolation, transport, etc.).
List Defeaters: Identify plausible differences (e.g., live retrieval vs. supplied text) that could break the chain.
Assign Evidence and Ownership: Determine who can produce the necessary evidence for each defeater.
Match Sampling to Claims: Ensure holdout units (e.g., templates, client files, time periods) match the claim's scope.
Define Observation and Loss: Specify exactly what is scored and what constitutes failure (including "red-line" outcomes).
Register Support/Defeat/Unresolved: Pre-define statistical criteria for each status.
Audit Interfaces: Check alignment and transmission for every adjacent link.
Separate Empirical Claim from Decision: Distinguish the evidence from the policy values and trade-offs.

3. Key Contributions and Results

A. The Legal-Research Case Study

The paper illustrates the audit using a hypothetical legal-research scenario. A developer reports high scores on a base model for supplied-text legal questions. A law firm intends to deploy this model in a retrieval-augmented workflow with human review.

Result: The audit reveals that the developer's benchmark (supplied-text) and the firm's local study (live retrieval + review) are parallel rather than compositional. The benchmark does not predict performance on live retrieval, and the local study does not validate the general capability of the base model.
Outcome: The local study can support a narrow authorization for specific request types (e.g., termination clauses) while defeating others (e.g., reasonable notice requests), but it cannot validate a broad "legal reasoning" claim. The benchmark score remains useful for screening but does not transfer to the deployment context without a bridging study.

B. Aggregation and Information Loss

The paper demonstrates how aggregation destroys interface evidence. Using a reanalysis of released model evaluations (Zhang et al., 2026) and simulations:

Stable Means, Unstable Items: A stable mean score ( $L \approx 0$ ) can mask significant item-level instability (high $INS$) or concentrated deterioration in the worst-performing tail (high $WTD$).
Cancellation Effects: Positive and negative changes can cancel out in an aggregate, hiding the fact that specific critical items have failed.
Implication: If a source report only publishes a total score, a downstream evaluator cannot determine if the source failures align with the features defining their target risk.

C. Distinctions in AGI Evaluation

The paper clarifies the roles of benchmarks in AGI evaluation:

Commensurability vs. Compensability: A weighted sum of scores is only meaningful if components are commensurable and compensability is acceptable for the specific decision.
Description vs. Prediction vs. Decision: A score can be a valid descriptive index without being a valid measure of a latent construct, or a valid predictor without being a causal mechanism.
Coverage vs. Reach: Adding more domains increases coverage but does not automatically establish unrestricted generality. The "Goodman problem" (rival extensions) remains; passing more tasks does not select the correct rule of extension.

4. Significance and Claims

The paper makes a modest but rigorous claim: it isolates a recurrent failure mode in distributed AI evaluation where separate studies warrant their own conclusions but fail to warrant the chain assembled from them.

Not a New Theory of Induction: The paper does not solve the problem of induction (Goodman's riddle) but provides a framework to make specific inductions inspectable and revisable.
Not a Replacement for Validity Theory: It operates within argument-based validity, adding a specific layer of scrutiny to the interfaces between inferences that existing frameworks often treat as implicit or automatic.
Diagnostic Utility: The primary value is diagnostic. It allows evaluators to identify "unsupported joins" in benchmark-to-use arguments. It does not guarantee that a system is safe or capable; rather, it clarifies the scope of what is currently supported by evidence.
Distributed Responsibility: The framework shifts the evidential burden. Developers are responsible for detailed source descriptions and lineage; deployers are responsible for context-specific sampling and workflow evidence. Neither party can unilaterally warrant the full chain without cooperation or bridging studies.

The paper concludes that a warranted benchmark-to-use argument must be a "declared, revisable chain" where endpoints align, assumptions are compatible, and uncertainty is propagated. Without this, the accumulation of successful evaluations does not constitute evidence for unrestricted general capability.