🤖 AI

A Control System, a Dataset, and a Recipe for Making Frozen LLM Agents Learn a Domain

This paper proposes a sample-efficient, auditable control system that optimizes a fixed, human-legible LLM agent harness using online reinforcement learning and multi-objective rewards, demonstrating its effectiveness across diverse task domains and model providers while releasing the code, datasets, and a deployment recipe for broader application.

Original authors: Debjyoti Paul

Published 2026-07-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Debjyoti Paul

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The AI Chef and the Magic Cookbook

Imagine you have a brilliant, super-smart chef who can cook almost anything. This chef is an Artificial Intelligence, specifically a Large Language Model (LLM). But here's the catch: this chef doesn't just wander into a kitchen and start cooking. They need a recipe, a set of tools, and a plan. In the world of AI, this setup is called a "harness." It's the instruction manual that tells the AI how to think, what tools to use (like a calculator or a search engine), and how to check its work before serving the final dish.

For a long time, scientists thought the only way to make the chef better was to train the chef themselves—teaching them new skills from scratch. But a newer idea suggests that maybe the chef is already great, and we just need to tweak the recipe. The big question researchers are asking is: Should we let the AI rewrite its own recipe while it's cooking (a risky, expensive move), or should we treat the recipe as a fixed menu of options and use a smart, simple system to pick the best one for the job? This paper dives into that question, testing whether a "frozen" AI (one that doesn't change its brain) can be supercharged just by changing how we give it instructions.

The Experiment: Taming the AI with a "Recipe Switchboard"

The authors of this paper decided to take a very careful, controlled approach. Instead of letting an AI go wild and rewrite its own code (which they argue is like letting a student rewrite the exam questions while taking the test), they built a "Control System." Think of this system as a giant, magical switchboard with exactly 729 different settings. Each setting is a specific combination of instructions:

Prompt Style: Should the AI be direct, structured, or reflective?
Tool Policy: Should it use tools only when needed, or always?
Memory: Should it remember past successes, or both successes and failures?
Planning: Should it just dive in, make a brief plan, or plan and revise?
Verification: Should it double-check its work at the end, or step-by-step?
Step Budget: How many steps should it take before stopping?

The researchers wanted to see if they could use a smart, learning system (called Reinforcement Learning) to flip these switches in real-time to find the perfect recipe for every task. They tested this on three different "kitchens":

Tool-Use: Making the AI manage a fake customer database (like a CRM).
Coding: Asking the AI to write code that passes specific tests (using the HumanEval benchmark).
Retrieval: Asking the AI to answer tricky questions by searching through a pile of documents (using HotpotQA).

They ran this experiment on two different "chefs": a local open-source model (Ollama) and a powerful cloud model (AWS Bedrock). They compared their smart, learning switchboard against two other methods: a random guesser (picking settings at random) and a "Static Baseline" (a single, pre-optimized recipe created by a tool called DSPy).

The Big Surprise: The "Static" Recipe Wins

Here is the twist that the authors didn't expect: The smart, learning switchboard didn't win.

Across almost every test, the "Static Baseline"—the single, pre-made recipe that was carefully crafted once and then left alone—performed just as well, or even better, than the system that was constantly trying to learn and adapt.

In the Tool-Use domain, the static recipe got 96% of tasks right on the Bedrock model, while the learning system only got about 62% right.
In Coding, the static recipe was just as good as the learning system, but it used far fewer "tokens" (the digital ingredients the AI eats), making it much cheaper and faster.
Even in the Retrieval domain, where things were harder for everyone, the static recipe held its own.

The authors realized why this happened. The "learning" system was trying to find the best setting out of 729 possibilities, but it didn't have enough time or attempts to learn the map. It was like trying to find the best path through a massive maze with only 25 tries; you're likely to get stuck in a dead end. The static recipe, however, was built using a method that only looked at plausible, high-quality options from the start, so it didn't waste time exploring bad paths.

What This Means for the Future

The paper concludes that for most practical, real-world situations, you shouldn't start with a complex, self-learning system that tries to figure everything out from scratch. Instead, you should:

Start with a strong, static recipe (like the one made by DSPy).
Only add the learning system later, and only if the tasks change so much that the old recipe stops working.

The authors also shared a "Recipe" (a guide called RECIPE.md) for other teams who want to build these systems. They emphasize that if you do use a learning system, you must be careful about "cold starts" (where the system picks a bad setting by accident at the beginning) and you need to watch out for crashes where one bad attempt ruins the whole batch.

In short, the paper suggests that while the idea of an AI that constantly rewrites its own instructions sounds cool and futuristic, in the real world, a well-written, fixed instruction manual often works better, faster, and cheaper than a system that is still trying to figure out how to write the manual while it's working. The "learning" part is useful, but only after you've already found a good starting point.

Technical Summary: A Control System for Learning Frozen LLM Agent Harnesses

Problem Statement
Production LLM agents are increasingly defined not by the underlying model, but by their "harness"—the surrounding architecture comprising prompt templates, tool sets, memory/retrieval layers, planning strategies, and verification policies. While recent 2026 systems like Meta-Harness and HyperAgents propose optimizing this harness via agentic proposers or self-modifying code, these approaches suffer from high costs, lack of auditability, and incompatibility with black-box APIs. This paper addresses the need for a constrained, auditable, and cost-effective method to optimize the harness without requiring the agent to rewrite its own code or access a coding-capable substrate.

Methodology
The authors propose treating the harness as a small, fixed, human-legible action space and learning a policy over it online using sample-efficient reinforcement learning.

Action Space: The harness is factored into six discrete levers:
1. Prompt Style: Direct, structured, or reflective.
2. Tool Policy: Never, when needed, or always (domain-reinterpreted).
3. Memory Policy: None, similar successes, or successes and failures.
4. Planning Policy: None, brief plan, or plan and revise.
5. Verification Policy: None, final check, or stepwise.
6. Max Steps: 2, 4, or 6.
  This yields $3^5 \times 3 = 729$ possible configurations.
Context Assembly: The system utilizes DSPy as the context assembler. For a selected configuration, DSPy constructs signatures and modules (e.g., ChainOfThought, draft-then-revise) and injects few-shot demonstrations based on the memory policy.
Reward Function: A multi-objective reward ( $R$ ) is calculated:
$R = w_s \cdot \text{success} + w_v \cdot \text{verifier\_score} + w_c \cdot \text{compliance} - w_u \cdot \text{unsupported\_claims} - w_\$ \cdot \text{tokens} - w_\ell \cdot \text{latency}$
Success is determined by domain-specific deterministic verifiers (unit tests for code, state diffs for tool-use, exact match for QA).
Controllers: The study compares four approaches:
1. Random: Uniform sampling over the 729 configurations.
2. $\epsilon$ -greedy Contextual Bandit: Linear value estimation with optimistic initialization to prevent cold-start failures.
3. REINFORCE: Softmax policy with entropy regularization and a moving-average baseline.
4. DSPy-Static Baseline: A single, fixed configuration optimized via DSPy's BootstrapFewShot on a small labeled subset, serving as the strongest non-adaptive benchmark.

Experimental Setup

Domains: Three verifiable task domains: Synthetic CRM tool-use workflows, HumanEval (code generation), and HotpotQA (multi-hop retrieval).
Models: Local Ollama (qwen2.5:7b) and AWS Bedrock (Haiku and Sonnet for spot checks).
Scale: 120 tasks (96 train, 24 test) across 4,620 logged episodes. Total cost was $7.62.

Key Results
Contrary to the hypothesis that online adaptive controllers would outperform static baselines, the DSPy-optimized static baseline consistently matched or exceeded the performance of the online controllers (Bandit and REINFORCE) across all domains and models.

Performance: In the Tool-use domain on Bedrock Haiku, the static baseline achieved 96% success at 285 tokens/episode, whereas the REINFORCE controller achieved 62% at 680 tokens/episode.
Sample Efficiency: The online controllers failed to converge to the static baseline's performance even when the episode budget was increased 5–12 times (300 episodes per run). The action space size ( $M=729$ ) proved too large for the available sample budget, creating a "sample-efficiency gap."
Cold-Start & Stability: The authors identified and fixed specific implementation pitfalls, including a cold-start trap where the bandit got stuck on a "no-tools" configuration due to deterministic tie-breaking, and a crash-isolation issue where a single malformed response could terminate an entire batch of jobs.

Contributions

A Control System: An implementation of online harness optimization using contextual bandits and REINFORCE over a 6-lever, 729-configuration space, built on DSPy.
A Dataset: A released suite of 120 verifiable tasks and 4,620 full trajectory/reward-decomposition logs across two model providers.
A Recipe: A provider-agnostic deployment playbook (RECIPE.md) that outlines a ten-step process for applying this control system, emphasizing bootstrapping with a static baseline before deploying online controllers.

Significance and Claims
The paper's primary significance lies in its empirical finding that online harness control is not a universal solution for small-to-moderate episode budgets. The authors argue that starting an online controller from a uniform prior over a large discrete space is inherently sample-inefficient compared to a targeted, metric-guided search (like BootstrapFewShot).

The paper concludes that the practical guidance for practitioners is not to replace static prompts with online controllers immediately, but to use the online controller after bootstrapping with a strong static baseline or when task distributions shift significantly enough to render static prompts stale. The work serves as a cautionary note against the "self-modifying code" trend, advocating instead for constrained, auditable, and sample-efficient policy learning over a fixed action space.

The AI Chef and the Magic Cookbook

The Experiment: Taming the AI with a "Recipe Switchboard"

The Big Surprise: The "Static" Recipe Wins

What This Means for the Future

Technical Summary: A Control System for Learning Frozen LLM Agent Harnesses

More like this