🤖 AI

Context Assembly as the Controlled Variable: A Control-Theoretic View of Harness Policies for Frozen LLM Agents

This paper proposes a control-theoretic framework for frozen LLM agents that treats context assembly as the controlled variable, utilizing an online outer policy to dynamically optimize prompt components while providing formal stability guarantees and uncertainty calibration.

Original authors: Debjyoti Paul

Published 2026-07-29

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Debjyoti Paul

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Art of the Perfect Prompt: A Control Theory Story

Imagine you are trying to teach a brilliant, but incredibly stubborn, robot how to solve a mystery. This robot is an AI, and it's "frozen," meaning its brain is set in stone; you can't rewire its neurons or teach it new facts. It knows a lot, but it doesn't know how to use that knowledge for your specific puzzle unless you give it the right instructions. In the world of Artificial Intelligence, this setup is called an "agent." For a long time, researchers have been trying to figure out how to make these agents reliable, treating them like machines that need to be steered. They use ideas from "control theory," which is basically the science of keeping things stable and on track—like how a cruise control system in a car constantly adjusts the gas pedal to keep the speed steady, or how a thermostat tweaks the heat to keep a room comfortable.

The big question this paper tackles is: What exactly should we be steering? Most people have been trying to control what the robot does next (like which tool to pick up) or who it talks to. But this paper suggests that the real magic happens before the robot even starts moving. It's about controlling the "context"—the specific instructions, the examples you show it, and the amount of background information you feed it. Think of it like a chef who can't change the recipe book (the frozen AI), but can change the ingredients, the lighting in the kitchen, and the notes on the counter to get the perfect dish. The authors want to know if we can build a smart system that automatically figures out the best way to set up these instructions on the fly, and if that system is stable enough to trust.

The Paper's Big Idea: Steering the Setup, Not the Robot

This paper, written by Debjyoti Paul in July 2026, proposes a fresh way to look at AI agents. Instead of trying to control the robot's actions directly, the authors suggest controlling the context assembly. In plain English, this means controlling the "setup" of the prompt: which template to use, which few examples to show the AI, how much information to retrieve from a database, and how many times the AI should check its own work.

The authors imagine a two-part system. The first part is the "frozen" AI agent—the robot that actually does the work. The second part is a "harness" or a controller sitting outside the robot. This controller is a smart learner (using methods like a "contextual bandit" or a "REINFORCE" policy) that constantly experiments with different setups to see which one gets the best results. It's like a stage manager who can't change the actor's lines (because the actor is frozen), but can change the lighting, the props, and the script notes to help the actor perform better.

What They Found: A Reality Check

The authors didn't just dream up this idea; they tested it to see if it actually works and if it's stable. They ran experiments with a specific setup where the controller had to choose from 729 different possible configurations (a combination of different prompt styles, tools, memory policies, and verification steps). They ran these tests for 60 episodes (trials) per run, totaling 240 episodes across different seeds.

Here is the honest truth about what they found:

1. The Stability Test (Is it getting better?)
The authors wanted to see if the controller was learning to get better over time, a concept they call "stability." In a perfect world, the controller's performance should go up or stay the same as it learns. However, their data showed something different. Over the 60 episodes they tested, the performance actually trended slightly downward or stayed flat for most runs. Three out of four test runs showed a tiny negative slope.

The authors explain that this isn't necessarily a failure of their idea, but a limitation of the test size. They argue that with 729 possible options to choose from, 60 episodes is simply not enough time for the controller to learn the ropes. It's like trying to learn a massive library of 729 different languages in just a few days; you haven't had enough time to practice enough to get good. They note that even with 300 episodes (five times more than their stability test), their companion paper found the system still couldn't beat a static, pre-set baseline. The conclusion is that the system is currently in an "under-sampled" regime—it's too small a sample size to prove the long-term theory, even though the theory itself makes sense.

2. The Confidence Test (Does it know when it's guessing?)
The second thing they checked was "uncertainty calibration." This asks: Does the controller know when it is confident and when it is unsure? In a real-world scenario, if the AI is unsure, it should tell a human to step in.

The results here were quite surprising and a bit messy. The controller's "confidence" score (based on how likely it thought its choice was) was extremely low—around 0.0014—while the actual success rate of the tasks was much higher, around 0.64 (64%) and 0.58 (58%). This is a huge gap, roughly two orders of magnitude.

The authors explain this isn't a bug in the learning; it's a math problem. Because there are 729 options, the probability of picking any single one is naturally tiny (about 1/729, or 0.0014). So, the controller looks "unconfident" all the time, even when it picks the right answer. This means that if you tried to use this raw confidence number to decide when to call a human for help, it wouldn't work—you'd call a human for every single task, even the easy ones. The paper suggests that to fix this, you would need to recalibrate the confidence signal (perhaps by looking at the gap between the top two choices rather than the raw probability), but they admit they haven't tested that fix yet.

The Bottom Line

This paper doesn't claim to have solved the problem of AI control. Instead, it offers a very specific, honest look at a new way of thinking: controlling the context rather than the action. It argues that while this is a promising and distinct idea, the current experiments show that we are hitting a wall because the space of possibilities (729 configurations) is too big for the amount of data we can realistically collect in a day.

The authors are careful not to call this a breakthrough. They explicitly state that their stability results are "noise-dominated" and that their confidence signals are currently "uninformative" without extra fixes. They are essentially saying, "Here is a new lever to pull, but right now, we don't have enough practice swings to know if it works perfectly, and our current way of measuring confidence is broken." It is a call for more data and better calibration methods, rather than a declaration of victory.

Technical Summary: Context Assembly as the Controlled Variable

Problem Statement

While the application of control theory to Large Language Model (LLM) agents is an established field as of mid-2026, existing literature primarily focuses on controlling specific discrete actions or system routing. Prior works have treated tool selection (Anonymous [2026]), action selection in massive discrete universes (Majumdar [2026]), or inter-agent message routing (Nogueira and Skogestad [2026]) as the controlled variables.

This paper argues that these approaches overlook a distinct, practically critical controlled variable: context assembly. This refers to the dynamic configuration of the prompt template, few-shot demonstrations, retrieved context volume, and the number of planning or verification passes. Unlike raw action streams or tool catalogs, context assembly is the specific set of parameters a harness engineer manually edits today. The paper posits that this variable is small enough to be enumerable yet complex enough to require online learning, and it can be optimized by a lightweight controller (contextual bandit or REINFORCE) sitting outside a frozen LLM.

Methodology

Formal Decomposition

The system is decomposed into two distinct policies:

Inner Policy ( $\pi_\theta$ ): A frozen LLM model that generates actions $a_t$ given a state $s_t$ and a context configuration $C_t$ . The model weights $\theta$ are never updated.
Outer Policy ( $\pi_\phi$ ): A learnable controller that selects the context configuration $C_t$ $C_{t}$ from a fixed, enumerable space $\mathcal{C}$ $C$ based on the task $x_t$ $x_{t}$ and interaction history $h_t$ $h_{t}$ .
- In the paper's instantiation, $|\mathcal{C}| = 729$ , factored across prompt style, tool/retrieval policy, memory policy, planning policy, verification policy, and step budget.
- The outer policy is parameterized either as a linear value function (for a contextual bandit) or a softmax policy (for REINFORCE).

Optimization Objective

The outer policy is optimized to maximize a multi-objective reward function while keeping $\theta$ fixed:
$\max_\phi \mathbb{E} [R_{task} + \lambda_v R_{verify} + \lambda_p R_{policy} - \lambda_c C_{tokens} - \lambda_\ell C_{latency} - \lambda_h P_{unsupported}]$
This objective balances task success, verification quality, and policy adherence against costs for token usage, latency, and unsupported operations.

Stability and Calibration Analysis

The paper evaluates the controller using two specific control-theoretic lenses:

Empirical Stability: Adopting the definition from Zhang et al. [2026], stability is defined as the non-decreasing expected reward over a sliding window as episodes accrue. This is a weaker, more measurable claim than Lyapunov asymptotic stability, suitable for online learning where the "state" is a set of learned weights.
Uncertainty Calibration: The paper analyzes whether the controller's confidence (defined as the softmax probability of the selected action) correlates with realized task success. This is critical for human-escalation scenarios where low confidence should predict failure.

Key Results

1. Empirical Stability

The authors conducted an instrumented experiment using 240 episodes across bandit and REINFORCE controllers on a tool-use domain (Ollama qwen2.5:7b).

Finding: The empirical results did not confirm the theoretical proposition of non-decreasing reward within the 60-episode-per-seed budget.
Analysis: Linear fits of reward against episode index showed slightly negative or flat trends for three of four runs. The authors attribute this not to a failure of the logic, but to the under-sampled regime. With an action space of 729 configurations, 60 episodes are insufficient for the running estimates to concentrate (a regime identified as $\Omega(M)$ in Majumdar [2026]). The results characterize the difficulty of learning in this space with practical, single-day experimental budgets rather than proving asymptotic convergence.

2. Uncertainty Calibration

The calibration analysis revealed a significant disconnect between the controller's confidence and its actual performance.

Finding: The Expected Calibration Error (ECE) was 0.607, a large miscalibration.
Mechanism: The mean confidence of the controller was approximately 0.0014 (roughly $1/729$ ), while the realized success rate was between 0.58 and 0.64.
Implication: Because the action space is large and scores are near-equally distributed, the raw softmax probability is bounded near $1/M$ . This makes the raw probability an uninformative signal for human escalation; essentially every episode appears "low confidence" regardless of success. The paper explicitly states that raw action-selection probability cannot be used as a confidence signal without recalibration (e.g., temperature scaling or value-margin definitions).

Significance and Claims

The paper makes a modest, specific claim: it identifies context assembly as a distinct controlled variable that has been underexplored compared to tool selection or agent routing.

Contribution: It provides a formal decomposition of the "frozen model + learnable harness" architecture and subjects this specific controlled variable to the same empirical scrutiny (stability and calibration checks) applied to other control-theoretic agent papers.
Honesty in Reporting: The paper emphasizes that its empirical results are "informative rather than flattering." It explicitly reports that practical episode budgets are insufficient to achieve the asymptotic stability or calibration required for deployment in their current form.
Design Point: Rather than claiming a solved problem, the paper flags the calibration issue as a genuine open design point. It suggests that for this approach to be viable, confidence signals must be redefined (e.g., via value margins) rather than relying on raw softmax probabilities over large discrete spaces.

In summary, the paper does not claim to have solved the stability of LLM harnesses. Instead, it rigorously demonstrates the sample complexity bottleneck inherent in learning context assembly over a large discrete space and highlights the calibration failure of naive confidence metrics, providing a realistic baseline for future work in this specific niche.