Technical Summary: ECHO - Prune to Act, Trace to Learn with Selective Turn Memory in Agentic RL

1. Problem Statement

Long-horizon language agents operating in tool-use environments (e.g., search, coding, deep research) face a fundamental bottleneck: the need to retain useful historical observations within a bounded context window while simultaneously identifying which specific past turns contributed to a successful final outcome for reinforcement learning (RL).

Existing context-management methods (truncation, summarization, retrieval) enable longer rollouts but introduce two coupled limitations:

1. Loss of Fine-Grained Evidence: As the number of turns grows, distant observations are either removed or collapsed into compressed states (e.g., rolling summaries). This makes it increasingly difficult for the policy to reuse specific, fine-grained evidence required for complex reasoning.
2. Loss of Traceability for Credit Assignment: Once original turns are no longer source-addressable (i.e., the raw tokens are replaced by a summary), outcome-based RL loses an explicit path to align policy updates with the specific evidence that supported a successful answer. Standard trajectory-level advantage assignment in methods like GRPO spreads credit densely across all generated tokens, reinforcing redundant searches, incidental reasoning, or summary-generation tokens alongside genuinely useful evidence-gathering steps.

This lack of source-level addressability leads to empirical failure modes where longer rollouts result in "turn proliferation," larger trajectory volumes, and slower training without proportional accuracy gains.

2. Methodology: ECHO

The authors propose ECHO (Selective Turn-Memory Framework), a system designed to jointly address history collapse and traceable learning through source-indexed reconstruction. ECHO operates on the principle of "Prune to Act, Trace to Learn."

2.1 Selective Turn-Memory Reconstruction

Instead of collapsing history into a single summary state, ECHO maintains a non-collapsing set of memory records.

• Source-Indexed Turn Memory: After each completed tool-use turn $u_i = (a_i, o_i)$, the agent generates a compact local finding $s_i$ and parses it into a memory record $e_i = (i, \alpha_i, m_i)$. Here, $i$ is the source-turn index, $\alpha_i$ is a compact rendering of the action/tool call, and $m_i$ is the parsed finding. This set $M_j$ remains linked to original environment turns even when raw observations fall outside the active context.
• Autoregressive Memory Selection: At compression boundaries, the policy is prompted to select a subset of historical memories useful for continuing the task. The selection context includes the current bounded state and a rendered list of source-indexed memories. The policy outputs source indices ($bI^{sel}_j$) of reusable evidence, constraints, or failed attempts.
• Bounded Context Reconstruction: The policy context for the next segment is reconstructed by rendering the selected source memories ($M_j[bI^{sel}_j]$) and appending recent local interactions, ensuring the context stays within the budget $B$.

2.2 Provenance-Guided Credit Assignment

ECHO utilizes the same source indices selected for context reconstruction to route outcome credit, creating a "provenance trace."

• Token-Level Hard Credit Mask: Unlike standard methods that assign the trajectory-level advantage to all tokens, ECHO constructs a mask $\mu^{(n)}_q$ that marks specific tokens as "credit tokens":
  1. Tokens in the final response segment.
  2. Action tokens from source turns selected into the final reconstructed context.
  3. Tokens corresponding to the "last-turn finding" of those selected source turns.
  4. All tokens generated during the memory-selection action spans.
• Positive Advantage Routing: For binary verifier rewards, the advantage is routed only through these credit tokens if the trajectory has a positive group-relative advantage ($A^{(n)}_+ = \max(A^{(n)}, 0)$). If the rollout is incorrect or below baseline, traceable credit is not assigned to avoid noisy supervision from potentially misleading selected turns.

3. Key Contributions

The paper formalizes and implements the following contributions:

1. Unified Reconstruction Interface: A formalization of context-managed multi-turn RL that separates history state management, bounded-context reconstruction, and source-level traceability. This framework explicitly highlights how collapsed-history methods sacrifice provenance needed for outcome-credit assignment.
2. ECHO Framework: A selective turn-memory method that stores completed turns as source-indexed memories and constructs bounded contexts via learned selection rather than global history collapse.
3. Provenance-Guided Credit Assignment: A mechanism that reuses selected source indices to route positive outcome advantage specifically to final answers, selected historical evidence turns, their memory findings, and the selection actions themselves.
4. Empirical Validation: Demonstrations that ECHO improves accuracy while reducing turn proliferation and trajectory volume compared to rolling-summary baselines, with consistent gains across both dense and Mixture-of-Experts (MoE) backbones.

4. Experimental Results

Experiments were conducted primarily on BrowseComp-Plus, a long-horizon tool-use QA benchmark, using the Qwen3-32B-Instruct and Qwen3-30B-A3B-Instruct (MoE) backbones.

• Performance on BrowseComp-Plus: ECHO achieved 43.4% held-out accuracy, significantly outperforming GRPO (28.9%) and the rolling-summary baseline SUPO (36.1%).
• Efficiency: While SUPO improved accuracy over GRPO, it suffered from rapid turn proliferation (62.5 turns vs. ECHO's 45.3) and a 4.18x trajectory split rate. ECHO maintained higher accuracy with fewer turns (45.3) and a lower trajectory volume (3.13 trajectories per rollout).
• Zero-Shot Generalization: ECHO demonstrated superior generalization across Multi-Objective QA, Code Generation, and Deep Information Seeking benchmarks. On the dense backbone, ECHO averaged 40.2%, outperforming SUPO (34.8%) and GRPO (33.6%). Similar trends were observed on the MoE backbone.
• Ablation Studies:
  • Replacing learned selection with static semantic top-k retrieval significantly reduced accuracy, confirming the necessity of policy-driven selection.
  • Removing the traceable credit mask (falling back to dense assignment) reduced both accuracy and stability, validating the importance of provenance-guided updates.

5. Significance and Claims

The paper claims that preserving provenance during context management is a practical mechanism to make long-horizon search more useful, rather than merely longer.

• Alignment of Acting and Learning: By using source-indexed reconstruction, ECHO aligns the evidence retained for future decisions with the evidence reinforced after successful rollouts. This resolves the tension between context efficiency (acting) and credit assignment (learning).
• Beyond Summarization: The results suggest that simply extending rollouts via summarization (as in SUPO) is insufficient and can lead to inefficient search behaviors. Traceable reconstruction allows the agent to prune history for acting while retaining the specific "breadcrumbs" needed for effective RL updates.
• Generalizability: The framework is not limited to a specific backbone architecture, showing robust performance across both dense and MoE models, and transfers effectively to diverse zero-shot domains.

The authors acknowledge limitations, noting that ECHO uses a "final-trace approximation" for credit assignment (not recursively following the full dependency chain of selected turns) and focuses primarily on text-based tool-use agents, leaving GUI and multi-agent settings for future work.