Technical Summary: Unified Context Evolution for LLM Agents

Problem Statement

Large Language Model (LLM) agents excel at solving multi-step interactive tasks by interleaving reasoning with environment feedback. However, a critical limitation persists: each task episode typically begins with a fixed context (system instructions, few-shot demonstrations), rendering any strategies or insights discovered during a specific episode ephemeral. Once the task concludes, these discoveries are lost.

Existing approaches to this "forgetting" problem fall into two categories, both with significant drawbacks:

1. Reinforcement Learning (RL): Methods that update model weights via gradient descent. While effective, they require access to model parameters and substantial computational infrastructure, making them inapplicable to closed-source LLMs.
2. Within-Task Adaptation: Methods like verbal self-critique (Reflexion) or iterative refinement (Self-Refine) that improve performance during a single task but discard improvements once the task ends.
3. Cross-Task Knowledge Libraries: Systems that accumulate knowledge across tasks but often store it in untyped pools (mixing all insights) or focus on a single knowledge form (e.g., only skills or only workflows). Furthermore, these systems often rely on one-shot evaluation to judge knowledge quality rather than tracking utility through repeated usage.

The core challenge addressed is how to maintain a persistent, evolving library of knowledge that distinguishes between different functional types of knowledge, tracks quality through usage, and dynamically allocates generation resources to the library's weakest areas, all without updating the underlying LLM weights.

Methodology: Unified Context Evolution (UCE)

The authors propose Unified Context Evolution (UCE), a gradient-free framework where the agent's context itself becomes the locus of learning. UCE externalizes agent experience into an evolving library of Evolvable Context Units (ECUs).

1. Evolvable Context Units (ECUs)

The library decomposes experience into four functionally complementary, typed knowledge entries:

• Memory: Factual statements about environment mechanisms discovered via trial-and-error within a single task instance (e.g., "A sinkbasin can clean items without toggling a faucet").
• Strategy: Conditional decision rules (IF-THEN) for branching points within a specific task type (e.g., "If 'put' fails, open the receptacle and retry").
• Workflow: Canonical execution sequences generalized for a specific task type (e.g., "Find target $\to$ Transform $\to$ Place in receptacle").
• Skill: Reusable multi-step sub-operations applicable across different task types (e.g., "Search receptacles, open containers, take target").

Each ECU is a tuple $(z_e, \tau_e, w_e, c_e, f_e)$, where $z_e$ is the type, $\tau_e$ is the applicable task type, $w_e$ is a retrieval condition, $c_e$ is the injected content, and $f_e$ is a fitness score.

2. Lifecycle and Retrieval

The framework operates in evolution cycles, each consisting of five phases:

• Evaluate: The agent runs on a held-out evaluation set using the current library. Success rates are recorded, but no fitness scores are updated to prevent data leakage.
• Collect: The agent runs on a disjoint collection set. Trajectories are stored, and fitness scores are updated based on outcomes.
• Schedule (Knowledge Yield Scheduling - KYS): A module analyzes the library's state (coverage gaps, fitness distributions) to allocate a generation budget ($B$) across the four ECU types. It prioritizes types where the library is weakest (e.g., uncovered task types or low average fitness).
• Generate: Type-specific generators distill candidate ECUs from collected trajectories. Crucially, generators use a reshuffle test to filter out instance-specific observations (e.g., "Potato is on Counter 1") in favor of generalizable mechanisms (e.g., "Fridge works when closed"). They also utilize quality feedback from high- and low-fitness ECUs to guide generation.
• Cleanup: Low-value or redundant ECUs are pruned. A Laplace-smoothed fitness score ($f_e = \frac{s_e + 1}{u_e + 2}$) tracks empirical contribution, where $s_e$ is cumulative outcome credit and $u_e$ is usage count.

Retrieval occurs at decision time. A retriever scores ECUs based on semantic similarity to the task description, a task-type match boost, and the ECU's fitness score. Top-$K$ ECUs per type are injected into dedicated sections of the agent's system prompt.

Key Contributions

1. Typed Knowledge Representation: The introduction of ECUs, which decompose agent experience into four distinct, functionally complementary types (Memory, Strategy, Workflow, Skill), each with independent generation conditions and retrieval boundaries.
2. Unified Context Evolution Framework: A complete lifecycle management system featuring Knowledge Yield Scheduling (KYS) to dynamically allocate generation resources and a usage-based fitness score to track quality through repeated application rather than one-time extraction.
3. Empirical Validation: Demonstration that UCE significantly improves agent performance on interactive benchmarks and that the accumulated knowledge library is transferable across different reasoning backbones (e.g., ReAct, Plan-and-Act, NoThinking) without retraining the models.

Experimental Results

The framework was evaluated on two text-based interactive benchmarks: ALFWorld (household tasks) and WebShop (online shopping).

• Performance Gains:
  • ALFWorld: UCE increased the overall success rate from 75.4% (ReAct baseline) to 96.3% at its peak. Significant improvements were observed in weaker task types, such as "Look" (50.0% $\to$ 100.0%).
  • WebShop: UCE improved the average task score from 45.1% to 61.3% and the purchase success rate from 30.0% to 42.0%.
• Transferability: The ECU library generated by UCE successfully transferred to alternative actor backbones. For instance, applying the UCE library to the "Plan-and-Act" backbone raised its success rate from 73.1% to 97.8%.
• Ablation Studies:
  • On ALFWorld, Workflow ECUs alone accounted for the majority of gains, suggesting procedural skeletons are highly effective for this domain.
  • On WebShop, no single ECU type matched the full library's performance. The shopping task required a combination of Workflow (procedural guidance), Strategy (recovery from navigation traps), and Memory (facts about page mechanisms), highlighting the necessity of the multi-type approach.
• Statistical Significance: Improvements over the ReAct baseline were statistically significant ($p < 0.05$) on both benchmarks.

Significance and Claims

The paper claims that UCE offers a practical, gradient-free alternative to reinforcement learning for improving LLM agents, particularly for closed-source models where weight updates are impossible. By treating context as a learnable, evolving entity, UCE bridges the gap between isolated task execution and persistent learning.

The authors emphasize that the framework's ability to distinguish knowledge types and track quality through usage addresses specific limitations in prior cross-task knowledge systems, which often treat all insights as equal or fail to adapt generation resources based on library maturity. The results suggest that for interactive tasks, the "what to know" (retrieved context) is as critical as the "how to act" (reasoning backbone), and that a structured, evolving library can effectively capture and reuse the "how to know" derived from experience.

The paper remains modest regarding future implications, noting that the current ECU types are hand-designed and that the framework's scalability to larger task sets or continuous action spaces (e.g., AndroidWorld) remains an open question. It also acknowledges that the current evaluation relies on specific model pairs and discrete action spaces, leaving the disentanglement of framework design versus generator capability as a subject for future investigation.