💻 computer science

ContractBench: Can LLM Agents Preserve Observation Contracts?

This paper introduces ContractBench, a deterministic benchmark revealing that current frontier LLM agents frequently fail to preserve the temporal validity and byte-level integrity of observation contracts (such as session tokens and URLs), a capability that does not scale monotonically with model size and requires specific failure-aware training to improve.

Original authors: Jicheng Wang, Yifeng He, Zili Wang, Hanwen Xing, Arkaprava De, Hao Chen

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jicheng Wang, Yifeng He, Zili Wang, Hanwen Xing, Arkaprava De, Hao Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Digital Handoff" Problem

Imagine you are hiring a very smart, but slightly clumsy, robot assistant to run errands for you. You tell the robot: "Go to the bank, get a temporary key, and use that key to open the safe."

In the real world, if the robot drops the key, changes the shape of the key, or tries to use the key after it has expired, the safe won't open. The robot failed, not because it didn't know what to do, but because it failed to handle the specific details of the handoff.

This paper calls these details "Observation Contracts."

The Contract: When a tool (like a bank API) gives the robot a piece of data (like a temporary password or a signed link), that data comes with two invisible rules:
1. The Expiration Date: You must use it now, or it becomes useless.
2. The "Do Not Touch" Rule: You must copy it exactly, letter-for-letter. If you change even one comma or space, it breaks.

The authors found that even the smartest AI agents today are terrible at following these rules. They often drop the key, melt it, or try to use it the next day.

The New Test: CONTRACTBENCH

To prove this, the researchers built a new test called CONTRACTBENCH. Think of it as a "driving test" for AI agents, but instead of driving a car, they have to navigate a digital obstacle course where every step requires passing a fragile, time-sensitive package to the next person.

The Course: It has 33 different scenarios (like downloading a file with a link that dies in 10 seconds, or logging in with a code that must be typed perfectly).
The Rules: The test is run by a strict computer program (no human judges). It uses a "virtual clock" to see if the agent is too slow, and a "digital fingerprint scanner" to see if the agent changed a single letter in the code.
The Score: If the agent breaks the time limit or messes up the spelling of the code, they fail.

What They Discovered (The Findings)

The researchers tested 38 different AI models (including the newest, most famous ones from OpenAI, Anthropic, Google, and open-source teams). Here is what they found, translated into everyday terms:

1. Even the "Smartest" Robots Fail

The Finding: No model passed 80% of the test. The best one (Claude Opus 4.6) only got about 78%.
The Analogy: Imagine the smartest student in the world taking a math test. They can solve complex calculus problems, but if you ask them to copy a 10-digit number perfectly without making a typo, they still get it wrong 22% of the time. Being "smart" doesn't mean being "careful."

2. The "Magic Cliff" (Qwen 3.5)

The Finding: In one specific family of models (Qwen 3.5), there was a sudden jump in ability. The small 4-billion-parameter model got 0% correct. The slightly larger 9-billion model jumped to 56%.
The Analogy: It's like a video game character who is useless until they reach Level 9. At Level 8, they can't even hold the key. At Level 9, they suddenly learn how to hold it gently. This wasn't a slow improvement; it was a sudden "awakening" of a specific skill called restraint (knowing not to change the data).

3. Bigger Isn't Always Better (The GPT-5 Rollercoaster)

The Finding: As the GPT-5 family of models evolved, their performance went up, then crashed, then went back up.
The Analogy: Imagine a chef getting a new recipe book.

Chef A (GPT-5): Makes a perfect cake.
Chef B (GPT-5.1): Gets a "smarter" version of the book but starts adding too much sugar and ruining the recipe (specifically, they started changing the ingredients instead of copying them).
Chef C (GPT-5.2): Fixes the recipe and goes back to making perfect cakes.
The paper suggests that making an AI "smarter" or "more conversational" can sometimes make it worse at following strict, boring rules.

4. The "Base" Models Can't Do It

The Finding: Models that haven't been "trained" to chat or follow instructions (Base models) got 0% on the test. Only models that had been taught how to follow instructions (Instruct models) could pass.
The Analogy: A raw lump of clay (Base model) can't hold a shape. You have to sculpt it (Post-training) to make it hold a cup of water. The ability to follow these contracts isn't natural; it has to be taught.

5. The "Hint" Works

The Finding: When the researchers told the AI exactly what it did wrong (e.g., "You changed a letter in the code"), the AI could fix its mistake and pass the test.
The Analogy: If you tell a child, "You dropped the ball," they might try again and catch it. But if you just say, "Try again," they might drop it again. The specific feedback was the key to fixing the error.

Why This Matters (According to the Paper)

The paper argues that we have been testing AI on whether they can "solve a puzzle" or "write a story." But in the real world, AI agents are often used to do jobs where precision is everything.

If an AI agent is managing your bank account, it can't afford to change a single digit in a transaction code. If it's managing a server, it can't afford to use an expired password.

The Conclusion: Current AI agents are like brilliant but clumsy interns. They understand the big picture, but they keep dropping the fragile tools they are handed. To make them reliable for real-world jobs, we need to stop just making them "smarter" and start teaching them to be more careful and more precise with the digital contracts they receive.

Technical Summary: CONTRACTBENCH

Problem Definition: Observation Contracts

Large Language Model (LLM) agents are increasingly deployed to execute multi-step workflows involving external APIs. While existing benchmarks evaluate whether agents can reach a final goal state, they often overlook a critical failure mode in production: the violation of observation contracts.

An observation contract is defined as a tool-returned artifact (e.g., a presigned URL, OAuth state parameter, session token, or HMAC-protected webhook) that carries two orthogonal constraints imposed by the external system:

Temporal Validity: The artifact must be used within a specific time window (Time-to-Live, TTL) before it expires or becomes stale.
Byte-Level Integrity: The artifact must be relayed byte-for-byte without alteration. Real-world agent stacks often introduce mutations through URL re-encoding, parameter reordering, truncation, or line-wrapping, which invalidate signatures or checksums.

Current benchmarks fail to jointly test these constraints. They typically treat tool outputs as mere information for the next decision, allowing agents to paraphrase or lose intermediate artifacts without penalty. In reality, a single violation of validity (using an expired token) or integrity (corrupting a signature) invalidates the entire workflow.

Methodology: CONTRACTBENCH

To address this gap, the authors introduce CONTRACTBENCH, a deterministic benchmark designed to measure observation-contract compliance.

Benchmark Design

Scope: 33 dual-axis tasks derived from real-world API contract patterns, including presigned URLs, OAuth flows, signed requests, HMAC webhooks, and rate-limit windows.
Dual-Axis Evaluation: Tasks are categorized into four quadrants based on the pressure they exert:
- Q1 (Control): Low validity/integrity pressure (sanity checks).
- Q2 (Validity): Isolates temporal planning (e.g., handling scheduled maintenance or rate limits).
- Q3 (Integrity): Isolates byte preservation (e.g., handling long tokens or URL traps).
- Q4 (Dual-Axis): Stresses both axes simultaneously, reflecting the prevalence of time-limited, signature-bound artifacts in production.
Execution Environment: Tasks run under a virtual clock to ensure full reproducibility. A FastAPI server emits deterministic failure labels based on real-world API specifications (e.g., AWS SigV4, OAuth 2.0, RFC 6585).
Evaluation Protocol:
- Deterministic Validators: No human raters or "LLM-as-a-judge." A programmatic validator consumes HTTP request logs to verify if every observation satisfies both the validity window and the integrity predicate (SHA-256 hash check).
- Failure Taxonomy: A 15-label taxonomy classifies failures into Validity (e.g., EXPIRED_BEFORE_USE, RATE_LIMITED), Integrity (e.g., MUTATED_TOKEN, SIGNATURE_MISMATCH), and Meta categories.
- Solvability: Every task includes an oracle solution that passes the validator, proving the task is solvable.

Experimental Setup

The authors evaluated 38 model variants across 15 families, including frontier proprietary models (Claude, GPT-5 series, Gemini) and open-source instruction-tuned models (Qwen, Mistral, Gemma, Llama, DeepSeek). Each model was run on the full 33-task suite with $k=3$ rollouts per task ( $n=99$ episodes per model) at temperature 0.

Key Findings

1. Frontier Models Fail Observation Contracts

No evaluated model clears 80% on the full suite. The top performer, Claude Opus 4.6, achieves 77.8%. This reveals that even state-of-the-art models struggle to preserve intermediate artifacts. Furthermore, ranking on CONTRACTBENCH does not correlate with general-purpose benchmarks; for instance, Gemini 2.5 Flash and GPT-5.1 underperform several open-source models, while Llama-3.3-70B scores near 0%.

2. Emergent Capability Cliff in Qwen 3.5

A sharp "capability cliff" is observed within the Qwen 3.5 family. Performance jumps from 0% at 4B parameters to 56.6% at 9B, scaling smoothly to 70.7% at 397B-A17B. Below the cliff, agents fail to exhibit mid-trajectory restraint (e.g., waiting for maintenance windows); above it, they demonstrate adaptive recovery. This suggests the capability is tied to specific training recipes rather than parameter count alone.

3. Post-Training is Necessary but Fragile

Necessity: Base models (non-instruction-tuned) across all families score 0%, indicating that contract compliance is a post-training capability, not a pre-training emergent property.
Fragility (Inverse Scaling): The GPT-5 family exhibits a non-monotonic "V-shape" regression. GPT-5.1 shows a significant drop in compliance (48.5%) compared to GPT-5 (70.7%) and GPT-5.2 (74.8%). This regression is structured: GPT-5.1 specifically re-introduces byte-level integrity failures (e.g., WRONG_VALUE, MUTATED_TOKEN), suggesting that post-training updates aimed at conversational helpfulness can erode specific inhibitions required for contract compliance (e.g., waiting before retrying, preserving bytes exactly).

4. Actionable Reward Signals

The authors tested whether failure labels could serve as in-context reward signals. Using 42 GPT-5.1 failures, they found that providing the correct failure label as a coaching hint improved pass rates by +7.1 percentage points compared to a wrong-label hint. This indicates that the taxonomy provides directional, actionable signals for test-time correction, particularly for integrity-style failures, whereas temporal failures often require architectural interventions (e.g., machine-readable TTLs) rather than simple prompting.

Significance and Contributions

The paper claims three primary contributions:

Formalization of Observation Contracts: The authors define observation contracts as a distinct class of agent failure governed by orthogonal temporal and integrity constraints, formalizing a problem previously overlooked in agent evaluation literature.
CONTRACTBENCH Benchmark: They provide the first deterministic, programmatic benchmark for this capability, featuring 33 tasks, a 15-label failure taxonomy, and a virtual-clock execution model that eliminates human/LLM judge ambiguity.
Empirical Insights into Agent Robustness: The study reveals that contract compliance is an emergent, non-monotonic capability that is neither guaranteed by general tool-use ability nor consistently improved by scaling. It highlights that "helpful" post-training can regress specific inhibitions required for reliable API interaction, and that current frontier models remain unsolved on this critical reliability metric.

The authors conclude that observation contracts must become a first-class evaluation target to ensure agents can act reliably under the inter-step constraints imposed by real external systems. They suggest that mitigating these failures may require architectural changes (e.g., handle-based artifact storage) rather than solely relying on larger or better-tuned models.