🤖 AI

Buried in Textual Debt: Context Pruning with Visual Evidence Preservation for MLLM Agents

The paper introduces SPARE, a KL-guided framework that effectively prunes redundant reasoning text in Multimodal Large Language Model agents while preserving essential visual evidence, thereby improving task accuracy and mitigating the "textual debt" problem in long-horizon multimodal tool-use scenarios.

Original authors: Yuchen Huang, Sijia Li, Jun Zhang, Yi R. Fung

Published 2026-08-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuchen Huang, Sijia Li, Jun Zhang, Yi R. Fung

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving field of artificial intelligence, a specific type of computer program has emerged that acts as a digital assistant capable of seeing and thinking. These systems, known as multimodal large language models, are designed to look at images, understand questions about them, and use external tools to find answers. To solve complex problems, they do not just guess; they break the task down into a series of steps, writing out their own thoughts and plans as they go. This process of writing down their reasoning helps them stay organized and coordinate different actions, much like a human might jot down notes while solving a puzzle. However, as these digital assistants work through long and difficult tasks, the text they generate begins to pile up. This accumulation of self-written notes eventually becomes so large that it crowds out the original image and the new visual clues the tools provide, causing the system to rely too heavily on its own past words rather than the fresh evidence in front of it.

Researchers at the Hong Kong University of Science and Technology have identified this phenomenon as "textual debt," a state where the history of the conversation drowns out the visual reality the agent is supposed to be analyzing. When an agent gets stuck on an early, incorrect idea, the sheer volume of text reinforcing that idea can blind it to new information, leading to mistakes. To solve this, the team developed a method called SPARE, which acts as a selective editor for the agent's memory. Instead of simply deleting old text or compressing images, SPARE carefully reviews the agent's past reasoning to decide what is still useful and what has become redundant. It works by asking a simple question: if the agent were to summarize its current situation in a few sentences, would it still need to read the specific paragraph of reasoning it wrote earlier to understand the next step?

The process involves a clever diagnostic check where the system compares two versions of its own thinking. In one version, the agent reads its full history of thoughts and actions. In the other, it reads that same history but with a compact summary of the current task state added to the mix. The system then measures how much the presence of that summary changes the agent's predictions for the next word. If the summary makes the agent's thinking shift significantly, it means the old paragraph contains unique, critical details that the summary missed, so the paragraph is kept. If the summary causes almost no change, it means the old paragraph is just repeating information that is already captured in the summary, making it safe to remove. This allows the system to prune away the "stale" text that clutters the context while preserving the specific visual details, such as coordinates or text found in an image, that are essential for the task.

To make this pruning even more effective, the researchers also trained the system to write better, more comprehensive summaries. By teaching the agent to condense its task state more efficiently, they enabled the pruning tool to remove even more of the redundant text without losing important information. When tested across several challenging benchmarks involving image editing, tool use, and visual search, this approach proved highly successful. The system managed to remove between 37.89% and 64.58% of the reasoning tokens—the words the agent wrote to itself—while actually improving its accuracy on the tasks. In many cases, the pruned agents performed better than those that kept their full history, suggesting that reducing the textual noise helped them focus more sharply on the visual evidence.

The findings challenge the common assumption that an agent needs to remember every single step of its reasoning to succeed. Instead, the study suggests that for long, complex tasks, the ability to forget the irrelevant parts of the conversation is just as important as the ability to reason. By clearing away the textual debt, the system restores its ability to attend to the image and the new data provided by its tools, rather than getting trapped in a loop of its own past assumptions. This work indicates that for artificial intelligence to become a truly effective long-term partner in visual tasks, it must learn not just how to think, but also how to let go of the thoughts that no longer serve the goal.

Technical Summary: Buried in Textual Debt: Context Pruning with Visual Evidence Preservation for MLLM Agents

Problem Definition: Textual Debt in Multi-Step Agents

Multimodal Large Language Models (MLLMs) deployed as multi-step agents rely on explicit reasoning to decompose tasks, coordinate tools, and interpret observations. However, this process generates a cumulative stream of self-generated text. Over long interaction trajectories, this "textual debt" can dominate the context window, causing the model to become increasingly conditioned on its own prior plans, descriptions, and assumptions rather than on the original visual input or new tool observations.

The authors identify two specific failure modes associated with this debt:

Redundancy: Once task-relevant visual evidence is successfully grounded, further reasoning text often becomes functionally redundant.
Misguidance: If early reasoning fails to attend to relevant visual regions, the accumulated text may elaborate on stale or incorrect hypotheses, actively misleading subsequent inference steps.

Existing solutions, such as visual token compression or generic summarization, do not adequately address this. Visual compression targets a modality that is often already under-attended, while generic summarization fails to distinguish between functionally necessary historical segments and redundant ones.

Methodology: SPARE

The paper proposes SPARE (Selective Pruning of Accumulated Reasoning with Visual Evidence Preservation), a post-hoc context-pruning framework designed to remove redundant reasoning while preserving critical visual evidence. SPARE operates at test time without requiring external verifiers or separate reward models.

Core Mechanism: Summary-Conditioned KL Divergence

SPARE reframes context compression as a "selective forgetting" problem. It utilizes a compact task-state summary ( $\sigma$ ) as a privileged diagnostic context. This summary is generated by the agent itself (triggered by an internal summarize_the_task tool) and contains the original question, completed tool calls, accumulated visual evidence, and remaining uncertainty.

The pruning process involves the following steps:

Replay and Comparison: For each candidate historical reasoning segment, the model is replayed under two contexts:
- Student Context ( $\rho_{stu}$ ): The original history without the summary.
- Teacher Context ( $\rho_{tea}$ ): The original history augmented with the compact task-state summary ( $\sigma$ ).
- Note: The "teacher" is the same model ( $\pi_\theta$ ) conditioned on the summary, not an external model.
Residual Sensitivity Measurement: The system computes a reverse-Kullback–Leibler (KL) divergence between the token-level continuation distributions of the student and teacher contexts.
- Low KL: Indicates the summary sufficiently covers the information in the segment; the segment is redundant.
- High KL: Indicates the segment contains information not captured by the summary (e.g., specific coordinates, OCR strings, or unique reasoning steps); the segment is critical.
Segment-Level Pruning:
- Segments with low residual sensitivity (low KL) are removed.
- Segments with high residual sensitivity are preserved. Crucially, these are not kept as raw text but are reconstructed into concise, structured visual evidence (e.g., "Crop: (1400, 900)", "Text: 'Stop'").
- A count-based threshold ( $\kappa$ ) is used rather than an average score to ensure that sparse but critical evidence tokens (like a single coordinate) prevent the pruning of an entire segment.

Enhancing the Summarizer via SFT

To enable more aggressive pruning, the authors employ Supervised Fine-Tuning (SFT) on the summarizer component. By training the agent to produce more compact and comprehensive task-state summaries (distilling knowledge from a larger model), the system can cover a broader fraction of the reasoning history. This expands the "safe-to-prune" region, allowing more redundant text to be removed without disrupting downstream decisions. Specifically, the authors collected 376 on-policy trajectories from Qwen3-VL-8B on VTC-Bench and prompted the stronger Qwen3-235B to produce compact task-state summaries. The Qwen3-VL-8B was then fine-tuned on the resulting (context, summary) pairs.

Key Contributions

SPARE Framework: A novel, evidence-preserving selective-forgetting method that uses summary-conditioned KL divergence to estimate the functional redundancy of prior reasoning. It prunes segments whose information is already consolidated in a task-state summary.
SFT-Enhanced Summarization: A strategy to fine-tune the summarizer, enabling more compact summaries that facilitate broader coverage and more aggressive pruning while retaining task-relevant information.
Empirical Validation: Demonstrations across multiple backbones (Qwen3-VL, Llama-3.1-Nemotron) and visual tool-use benchmarks (GTA, VTB, MNMS, etc.) showing that SPARE achieves the highest average accuracy among pruning methods while significantly reducing context length.

Experimental Results

The authors evaluated SPARE on five benchmarks requiring multi-step reasoning and tool invocation: GTA, V*, BLINK-Jigsaw, VisualToolBench (VTB), and m&m's (MNMS).

Performance vs. Pruning: SPARE consistently achieved the highest average accuracy among all pruning methods.
- On Qwen3-VL-30B, SPARE improved average accuracy from 44.30% (Full Trace) to 49.19% while pruning 63.70% of reasoning tokens.
- On Qwen3-VL-8B, it maintained comparable performance (53.47% vs. 54.27%) with 37.89% pruning.
- On Llama-3.1-Nemotron-Nano-VL-8B, it improved accuracy on specific benchmarks (GTA, B-Jigsaw, VTB) and nearly matched the Full Trace accuracy under 64.58% pruning.
Comparison to Baselines: SPARE outperformed "Delete All Reasoning" (which removes all text) and "Visual Evidence-Only" (which non-selectively rewrites all text). This confirms that selective pruning based on residual sensitivity is superior to non-selective compression.
Ablation Studies:
- Adaptive Triggering: Allowing the model to decide when to summarize (Model-selected) outperformed fixed-interval or pre-answer summarization.
- Component Necessity: The combination of KL-based selection and evidence reconstruction yielded the best results; using either component alone resulted in lower accuracy or insufficient compression.
- SFT Impact: Fine-tuning the summarizer resulted in a relative increase of 23.8% in pruned tokens on VTB and 13.5% on MNMS compared to the base summarizer, while improving task accuracy on both benchmarks.

Significance and Claims

The paper claims that reducing "textual debt" is not merely a computational optimization but a grounding mechanism. By pruning stale, self-generated text, SPARE restores the model's reliance on visual evidence and mitigates over-conditioning on its own linguistic history.

The authors position SPARE as a test-time solution that requires no auxiliary models or training data beyond the agent's own trajectories. They argue that inference-time selective forgetting is an effective mechanism for long-context multimodal reasoning, allowing agents to maintain progress without being buried in their own accumulated text. The code is made available to facilitate further research in this direction.