💬 NLP

LLM Agents Are Latent Context Managers: Eliciting Self-Managed Context via a Proprioceptive Dashboard

This paper introduces VISTA, a training-free, model-agnostic dashboard that exposes an agent's internal context state (such as token usage and recency) to the model itself, enabling capable LLMs to effectively self-manage long-horizon contexts and significantly improving performance on complex benchmarks without requiring learned compression policies.

Original authors: Binyan Xu, Haitao Li, Kehuan Zhang

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Binyan Xu, Haitao Li, Kehuan Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Problem: The "Blind" Librarian

Imagine you have a brilliant librarian (the AI Agent) who is trying to solve a complex mystery. To do this, the librarian needs to read thousands of pages of clues, emails, and reports.

However, the librarian has a very specific problem: They are "proprioceptively blind."

Proprioception is your body's ability to know where your limbs are without looking at them. You know your hand is raised without staring at it.
The AI's Blindness: The AI can read the words in its memory, but it cannot "feel" the size or age of those words. It doesn't know if a specific chunk of text is a tiny note (0.3KB) or a massive encyclopedia entry (20KB). It doesn't know how long ago it read something or how many times it has looked at it.

Because the AI is blind to these details, it often makes bad decisions. It might try to summarize a tiny, important note (wasting time) or delete a huge, crucial file because it thinks it's just "old junk."

The Old Solutions: The "Blind" Cleanup Crews

Before this paper, researchers tried to fix this in two ways, both of which had flaws:

The System Manager (The "Blind Janitor"): The computer system automatically deletes old files or hides them behind a curtain. The AI doesn't know what was deleted or why. If the AI needs that file later, it's gone forever.
The Learned Policy (The "Trained Intern"): They train the AI to learn a rule like "always delete the oldest file." But this is risky. Sometimes the oldest file is the most important one. Also, this requires training the AI specifically for that job, which doesn't work well on new types of tasks.

The New Solution: VISTA (The "Dashboard")

The authors introduce VISTA (Visible Internal State for Tool Agents). Think of VISTA as giving the librarian a high-tech dashboard and a magic storage room.

1. The Dashboard (The Proprioceptive View)

Instead of just handing the AI a stack of papers, VISTA gives it a digital dashboard that shows:

Token Count: "This file is 20,000 words long."
Recency: "You read this 10 minutes ago."
Access History: "You've looked at this 5 times."
Budget: "You have 50,000 words of space left."

Now, the AI isn't guessing. It can see exactly how "heavy" each piece of information is.

2. The Magic Storage Room (Lossless Archiving)

When the AI sees a file is too heavy for its current workspace, it doesn't just delete it. It puts it in a Magic Storage Room.

Lossless: The file is stored exactly as it was, byte-for-byte. Nothing is summarized or lost.
Addressable: The dashboard gives the AI a specific "key" or "handle" (like a file path) to that storage room.
Recoverable: If the AI needs that heavy file later, it can use the key to pull the exact original file back into its workspace instantly.

How It Works in Practice

Imagine the AI is working on a task that requires reading a massive email thread (20KB) and a tiny confirmation link (0.3KB).

Without VISTA: The AI might try to summarize the email (losing details) or delete it because it's "old." Later, when it needs a specific number from that email, it fails because the summary was wrong or the file is gone.
With VISTA:
1. The AI looks at the dashboard and sees: "The email is huge (20KB), the link is tiny (0.3KB)."
2. The AI decides: "I need to save space. I will move the huge email to the Magic Storage Room and keep the tiny link here."
3. The dashboard updates to show: "Email is Archived. Key: archive/email_01.json."
4. Later, the AI needs a specific detail from that email. It looks at the dashboard, sees the key, and says: "Retrieve archive/email_01.json."
5. The exact file reappears, and the AI solves the problem perfectly.

The Results: Why It Matters

The paper tested this on several difficult benchmarks (LOCA-Bench, BrowseComp-Plus, GAIA) using different AI models (Gemini, Claude, etc.).

No Training Needed: They didn't have to retrain the AI. They just added the dashboard and the storage room.
Big Gains: On one test (LOCA-Bench), they improved a model's success rate from 22.7% to 50.7%.
Works Everywhere: It worked on small tasks (10K tokens) and massive tasks (millions of tokens).
The "Dashboard" is Key: When they removed the dashboard but kept the storage room, performance dropped. This proves that the AI needs to see the data about the data (the metadata) to make good decisions.

The Bottom Line

The paper argues that AI agents aren't "bad" at managing their memory; they are just blind to the state of their own memory.

By giving them a dashboard to see the size and age of their memories, and a lossless archive to store and retrieve them perfectly, we can unlock their ability to handle long, complex tasks without needing to teach them new rules or delete important information. It turns a "blind" agent into a self-aware manager.

Technical Summary: LLM Agents Are Latent Context Managers

Problem Statement

Long-horizon tool agents face a fundamental bottleneck: their working memory (context) grows until it crowds or overflows the model's context window. Existing approaches to managing this growth fall into two categories, both of which the authors argue leave a critical gap:

System-Controlled Management: Systems like OS-style paging or stale-observation masking manage context beneath the agent. While they track statistics (size, age, usage), the agent cannot inspect this state. Decisions are made by fixed rules that cannot predict which evidence will be critical later.
Agent-Controlled (Learned) Management: Methods like "Context-as-a-Tool" or reinforcement learning-based compression policies move the decision into the agent. However, these often rely on summarization or deletion, which discards exact evidence. Furthermore, they are tied to specific training settings and backbones.

The core hypothesis of this work is that frontier Large Language Models (LLMs) are proprioceptively blind to their own context. From the prompt alone, an agent cannot perceive the size, age, access history, or remaining budget of its context blocks. The authors argue that competent context management is already latent in capable models (acquired via pretraining on note-taking and retrieval traces), but the bottleneck is a missing interface to expose this state, not a missing policy.

Methodology: VISTA

The authors introduce VISTA (Visible Internal State for Tool Agents), a training-free, model-agnostic layer designed to elicit self-managed context. VISTA operates through three main components:

1. Context Stream and Block Representation

VISTA rewrites the raw interaction transcript into a stream of addressable, typed blocks. Each block (user message, tool call, tool result) is assigned a stable identifier, type, token estimate, and state.

Visible Blocks: Sent to the model in the active prompt.
Archived Blocks: Replaced by compact handles (path, checksum, summary) while the exact byte payload is stored externally in a "hidden trajectory."
Blocked/Deleted Blocks: Oversized results that cannot fit are either blocked (with a notification) or explicitly deleted.

2. The Proprioceptive Dashboard

At every step, the agent receives the visible context stream plus a dashboard. This dashboard is not a learned oracle but a runtime report of the workspace state ( $W_t$ ), including:

Budget Bar: Current token usage vs. limit.
Block Metadata: ID, estimated token count, age/recency, type, and status (visible, pinned, archived, blocked).
Access History: Tracks which blocks have been recently accessed.

This dashboard provides the "proprioceptive" signals (size, cost, recency) that the agent needs to make keep-or-drop decisions but cannot infer from text alone.

3. Meta-Context Tools

The agent is equipped with tools to act on the dashboard:

archive(block_id, summary): Removes a block from the active prompt, stores the exact payload externally, and returns a handle.
recover(path): Retrieves the exact byte-identical payload from external storage.
Mode Switching: If the prompt exceeds the budget, the system enters "overflow mode," disabling ordinary tool calls until the agent reduces the visible context (usually by archiving).

Theoretical Guarantee: The paper proves (Proposition 1) that under budget pressure, lossless recovery is necessary. A non-recovering method (deletion, summarization) with a bounded information state cannot guarantee the retrieval of specific, unknown future evidence bits with high probability, whereas VISTA can achieve probability 1 if the recovered block fits the budget.

Key Contributions

Framing Context Management as a Meta-Tool Decision: The paper identifies "context proprioception" as the missing interface. It argues that LLM agents cannot read their own context state (size, budget, history) from the prompt, necessitating an external interface.
VISTA Implementation: A training-free, model-agnostic layer that pairs a per-block metadata dashboard with lossless archive and recovery mechanisms. It isolates the "what to expose" question from the "how to act" question.
Empirical Validation: Demonstrating that the same untrained interface improves performance across different trajectory scales (10K to 1M tokens) and diverse model backbones without fine-tuning.

Experimental Results

The authors evaluated VISTA on LOCA-Bench (million-token stress test), BrowseComp-Plus (100K-scale retrieval), and GAIA (10K-scale general assistant).

Performance Gains: On LOCA-Bench, VISTA improved the success rate of Gemini-3-Flash from 22.7% to 50.7% and Claude-Sonnet-4.5 from 8.0% to 34.7%. It outperformed ReAct, deletion/masking baselines, and compression-based methods (SLIM, Context-Folding).
Cross-Backbone Transfer: The same untrained VISTA layer improved performance across four different backbones (Gemini-3-Flash, Claude-Sonnet-4.5, DeepSeek-V4-Pro, GLM-5), suggesting the gains are due to elicitation of latent skills rather than model-specific tuning.
Ablation Studies:
- Dashboard is Critical: Removing the dashboard (leaving archive/recovery tools but no state view) caused a significant drop in accuracy (e.g., from 50.7% to 37.3% on LOCA-Bench), confirming that the agent needs the state interface to use the tools effectively.
- Lossless Recovery is Necessary: Removing the recovery capability (turning archive into simple clearing) also degraded performance, validating the theoretical necessity of exact payload preservation.
Pressure Sensitivity: As context growth pressure increased (from 8K to 256K tokens), the performance gap between VISTA and baselines widened, indicating that the proprioceptive interface becomes more valuable as context management becomes harder.
Offline Transfer: On AMA-Bench (offline trajectory memory), VISTA achieved higher F1 scores than specialized memory adapters, demonstrating its utility as a general workspace representation.

Significance and Claims

The paper claims that context management is an interface problem, not solely a compression or policy problem.

Elicitation over Training: The results suggest that capable models already possess the latent ability to manage context if the necessary runtime state (size, budget, history) is made visible. The dashboard acts as a "proprioceptive" channel that unlocks this latent competence without requiring fine-tuning or learned policies.
Complementarity: VISTA is positioned as complementary to post-training. While training improves how an agent acts on information, VISTA supplies the information that is otherwise absent from the prompt.
Modesty: The authors acknowledge limitations: VISTA does not guarantee the agent will use the dashboard correctly (a model can still misread it), and it does not replace the need for models with latent context-management skills. The gains are most pronounced in models with sufficient latent capability (e.g., GLM-5 saw smaller gains).

In summary, VISTA demonstrates that making the hidden runtime state of an agent's context visible via a dashboard, paired with lossless recovery, allows untrained models to self-manage long-horizon tasks more effectively than systems relying on fixed rules or learned compression policies.