🤖 AI

Agentic Context Learning with Self-Discovered Specification

This paper identifies that the primary challenge in context learning for large language models is not merely accessing content but acquiring implicit, task-specific specifications distributed across the context, and demonstrates that a simple intervention called PSCI, which extracts and enforces these specifications through adversarial checking, significantly outperforms existing baselines on the CL-Bench benchmark.

Original authors: Jike Zhong, Ming Li, Yuxiang Lai, Ziyan Yang, Jingyu Xie, Jihyung Kil, Zheda Mai, Shao-Yuan Lo, Ren Xiang, Konstantinos Psounis, Yuanyuan Lei

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jike Zhong, Ming Li, Yuxiang Lai, Ziyan Yang, Jingyu Xie, Jihyung Kil, Zheda Mai, Shao-Yuan Lo, Ren Xiang, Konstantinos Psounis, Yuanyuan Lei

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've just been handed a massive, 40,000-page instruction manual for a brand-new, super-complex video game. The manual is full of lore, character backstories, and world history. Then, your friend asks you a simple question: "What happens if I skip a level?"

You might think the hardest part is finding the answer in that giant book. You'd think, "If I just search for 'skip level,' I'll find the rule!" But here's the twist: the paper argues that for modern AI models, the real problem isn't finding the answer (the content). The real problem is finding the rules of the game that tell you how to write that answer correctly (the specifications).

The "Near-Miss" Mystery

The researchers tested this on a benchmark called CL-Bench, which contains 1,899 tricky tasks where the AI has to learn new rules from long texts. The results were surprising. Even the smartest AI models (like GPT-5.1) only got about 22.55% of the tasks perfectly right.

But when they looked closer at the mistakes, they found something weird. The AI wasn't usually getting the facts wrong. It was getting the format wrong.

The Fact: The AI knew the missing ID was "404."
The Rule: The manual said, "If you see a missing ID, you must write the exact flag Sequence_Gap_Warning."
The Mistake: The AI wrote "404" but forgot the flag.

It was a "near-miss." The AI knew the content but missed the local specification. In fact, the paper found that 55.4% of all the rules in the test were about these specific behaviors (formats, ordering, exact phrases), while only 22.6% were about factual content.

Why "Searching" Didn't Work

A natural guess would be: "Maybe the AI just can't find the right page in the manual." So, the researchers tried 12 different methods to help the AI search better, like summarizing the text or pulling out relevant chunks.

The result? None of them worked well. The best of these search-heavy methods still only scored around 23%. This suggests that simply finding the right page isn't the issue. The problem is that the rules are often hidden in the background—like a footnote in a paragraph about "event semantics"—and the AI doesn't realize it needs to follow them unless it explicitly looks for them.

The "Private Contract" Solution

To prove this, the researchers designed a simple trick called PSCI (Private Specification-Contract Induction). Think of it like this:

Before the AI tries to answer the question, they force it to do a "pre-game meeting."

Induce: The AI reads the whole manual and writes down a "Private Contract" of all the hidden rules it found (e.g., "Rule 1: Always use the flag Sequence_Gap_Warning").
Generate: The AI writes its answer, but it has to follow that contract.
Check: A "referee" (another AI step) checks the answer against the contract. Did you use the flag? Did you follow the order?
Repair: If the referee says "No," the AI fixes the answer before showing it to you.

The result? This simple four-step process boosted the GPT-5.1 score from 22.55% to 28.14%. That's a 5.59 percentage point jump, which is huge in this world. It also worked on other models like Qwen3.5-27B (jumping from 14.14% to 19.42%) and Gemini 3 Pro (jumping from 16.14% to 22.31%).

What This Rules Out

The paper is very clear about what doesn't work:

Just searching more: Throwing more search tools at the problem didn't help.
Just thinking harder: Using the "GPT 5.1 (High)" setting from the original CL-Bench evaluation only got the model to 23.7%, which is still lower than the 28.14% achieved by the PSCI trick.
Generic checklists: If you just ask the AI to "check your work" without first making it write down the specific rules from this context, it doesn't help. The rules must be specific to the task at hand.

How Sure Are They?

The authors are quite confident in their findings because they tested this rigorously. They didn't just guess; they ran 17 different "ablation" experiments (breaking the method apart to see what works).

They proved that if you shuffle the rules (give the AI the wrong contract), the score crashes back down to 19.80%, proving the rules must match the specific task.
They proved that if you skip the "contract" step and just ask the AI to check its own work later, it fails. The contract must be written before the answer is generated.
They even had human experts check 100 of the AI's answers. The humans agreed with the computer's judgment 96% of the time, confirming the AI was actually getting better, not just guessing.

The Bottom Line

The paper suggests that for AI to learn new rules from long texts, it can't just be a good librarian who finds the right book. It needs to be a good lawyer who reads the fine print, writes down a contract of the rules, and then signs that contract before doing any work.

While the score of 28.14% is the best so far, the authors note it's not a "solved" problem yet. There's still a lot of room for improvement, especially in the "enforcement" part—making sure the AI actually follows the contract it wrote for itself. But the main takeaway is clear: Context learning isn't just about finding the answer; it's about discovering the rules of the game first.

Technical Summary: Agentic Context Learning with Self-Discovered Specification

1. Problem Definition: The Context Learning Bottleneck

The paper addresses Context Learning, an inference-time task where Large Language Models (LLMs) must learn and apply novel, task-specific knowledge from intricate, long contexts absent from pre-training. Unlike traditional long-context tasks (e.g., document understanding) which focus on retrieving relevant facts, context learning requires the model to operate within a newly defined local environment (e.g., a product manual, regulatory code, or API spec) where the rules of validity are often implicit.

Despite strong performance on general language tasks, frontier models struggle significantly in this setting. On CL-Bench (a benchmark specifically designed for context learning with 1,899 tasks), even the best models score below 24% strict task success. The authors hypothesize that failures are not merely due to an inability to access content within long contexts, but rather a failure to acquire local specifications—the behavioral constraints, formats, edge-case handling rules, and completeness conditions distributed across the context that determine whether an answer is valid.

2. Empirical Analysis and Diagnosis

The authors conducted a comprehensive empirical study on CL-Bench to diagnose the root cause of these failures:

Failure Mode Analysis: While the strict task pass rate is low (22.55%), the rubric-level pass rate is much higher (76.01%). This indicates that models often satisfy most requirements but fail on a critical few. Among tasks failing by exactly one rubric item, 75.3% of the missed items were specification items (compliance with local behavioral constraints) rather than factual content errors.
Taxonomy of Requirements: Across 31,592 rubric items, 55.4% evaluate specification acquisition (learning local rules), while only 22.6% evaluate content acquisition (retrieving facts).
The Query-Context Asymmetry: A key finding is that 76.7% of specification items are not inferable from the user query alone; they are distributed across the surrounding context (e.g., in examples, validation patterns, or repeated structures). However, 95.5% of these specifications are traceable to the context, meaning they are learnable obligations rather than hidden evaluator preferences.
Limitations of Existing Methods: The authors evaluated twelve baselines spanning retrieval, context restructuring, self-correction, and iterative retrieval. None exceeded the direct full-context prompting baseline by more than one percentage point. This suggests that methods focused on surfacing query-relevant content fail because they miss the validity-relevant specifications that govern the response's form and procedure.

3. Methodology: Private Specification-Contract Induction (PSCI)

To validate the diagnosis that specification acquisition is the bottleneck, the authors propose PSCI, a deliberately simple, training-free, test-time scaffold. PSCI operates in four sequential steps:

Specification Discovery (Induction): The model induces a private specification contract ( $K$ ) from the task ( $x$ ) and context ( $c$ ). This contract is a structured set of obligation items (formats, validation checks, ordering constraints) required for local validity. Crucially, this contract is induced before generation and is never observed by the public rubric.
Generation: The model generates an initial draft answer ( $y_0$ ) conditioned on the task, context, and the induced contract $K$ .
Adversarial Checking: An adversarial checker audits $y_0$ against $K$ (not the public rubric), producing a structured diagnosis ( $d$ ) identifying which obligations are satisfied, partially satisfied, or violated.
Repair: A repair step patches the violations identified in $d$ to produce the final answer ( $\hat{y}$ ), preserving correct content while enforcing the contract.

The core innovation is treating the specification contract as a control object that shapes generation, rather than a post-hoc grader.

4. Key Results

The authors evaluated PSCI on CL-Bench using GPT-5.1, Qwen3.5-27B, and Gemini 3 Pro:

State-of-the-Art Performance: PSCI achieved a strict pass rate of 28.14% with GPT-5.1 (Medium), an absolute gain of +5.59 percentage points (24.8% relative improvement) over the direct baseline (22.55%). This performance surpassed even GPT-5.1 (High) from the original CL-Bench evaluation (23.7%).
Cross-Model Generalization: The gains were consistent across models:
- Qwen3.5-27B: Improved from 14.14% to 19.42% (+5.28 pp).
- Gemini 3 Pro: Improved from 16.14% to 22.31% (+6.17 pp).
Rubric-Level Gains: PSCI improved the global rubric pass rate by +5.12 pp. Notably, the improvement was larger for specification items (closing 23.2% of the gap) than for content items (closing 15.5%), confirming the method targets the identified bottleneck.
Comparison to Baselines: All twelve alternative baselines (including Self-RAG, Reflexion, and Iterative Retrieval) remained within 0.78 pp of the direct baseline, failing to match PSCI's performance.

5. Ablation Studies and Mechanism Validation

Seventeen ablations were conducted to isolate the role of task-specific specifications:

Necessity of Task-Specificity: Shuffling contracts (using a contract from a different task) caused performance to collapse below the baseline (19.80%), proving that specifications must be aligned with the specific task context.
Grounding in Context: Zero-shot context-derived specifications improved performance, but generic static specifications did not, confirming that specifications must be extracted from the current context.
Enforcement vs. Discovery: Discovery alone (inducing the contract without checking/repair) reached 26.09%, but the full pipeline (28.14%) showed that enforcement is critical.
Control Object vs. Post-Hoc Grader: Inducing specifications after generation (Answer-first ordering) failed (23.67%), demonstrating that the contract must act as a control object to shape the generation process, not just a repair guide.
Compute Efficiency: Compute-matched controls (e.g., generic self-reflection, repeated static specs) failed to match PSCI, indicating the gain comes from the structure of the computation (specification discovery and enforcement), not merely increased inference cost.

6. Significance and Claims

The paper claims three primary contributions:

Diagnosis: It identifies specification acquisition—inferring query-implicit but context-traceable local obligations—as the key bottleneck in context learning, distinct from content acquisition.
Methodology: It introduces PSCI, a simple, training-free scaffold that achieves state-of-the-art results by explicitly inducing and enforcing these local specifications.
Mechanism: It demonstrates through extensive ablation that context learning requires a two-stage process: specification discovery (grounded in the context) and specification enforcement (via checking and repair).

The authors conclude that context learning hinges not just on retrieving the right facts, but on acquiring and adhering to the local "rules of the game" defined within the prompt. They note that while PSCI significantly improves performance, enforcement remains a residual bottleneck, as models often surface the correct obligations but fail to consistently execute them in the final output. The work suggests that future progress in context learning requires mechanisms that better internalize and enforce these local validity conditions.