ExpWeaver: LLM Agents Learn from Experience via Latent RAG

Technical Summary: ExpWeaver: LLM Agents Learn from Experience via Latent RAG

Problem Statement
Integrating past interactions as reusable experiences is a critical approach for enhancing Large Language Model (LLM) agent planning and reasoning. However, existing methods are largely confined to explicit text space. Current approaches typically store experiences as textual summaries and retrieve them via semantic similarity, concatenating them into the model's context window. This paradigm introduces significant inefficiencies:

1. Token Overhead: Explicit concatenation consumes substantial context window space, limiting the number of retrievable experiences and increasing inference costs.
2. Architectural Decoupling: Most methods employ a separate Retrieval-Augmented Generation (RAG) module (e.g., embedding-based retrievers or search engines) that operates independently from the LLM's internal representations. This separation prevents joint optimization of retrieval and generation toward the downstream task objective.
3. Representational Bottlenecks: Text-level retrieval relies on surface semantic similarity, which may fail to capture deeper structural correspondences in problem-solving strategies required for complex reasoning or coding tasks.

Methodology: ExpWeaver
To address these limitations, the authors propose ExpWeaver, a framework that enables LLM agents to learn from experience via latent retrieval-augmented generation. Unlike prior work, ExpWeaver performs retrieval directly within the LLM's latent space without requiring a separate RAG module. The framework consists of three core components:

1. Experience Bank with On-Policy Alignment:

   • Past trajectories (query, reasoning trace, output, reward) are compressed into textual summaries and encoded into dense latent embeddings ($z$) using the same LLM ($p_\theta$) used for generation.
   • Specifically, the embedding is derived from the hidden state of the last token at the final transformer layer ($z = h_\theta(s)$).
   • This design ensures on-policy alignment: as the policy evolves during training, recent experiences naturally reside in a compatible embedding region, creating an implicit recency bias analogous to on-policy reinforcement learning.

2. Latent Retrieval-Augmented Generation:

   • Latent Retrieval: At each decoding step $t$, the current hidden state $h_t$ serves as the query to retrieve a top-$K$ set of experience embeddings ($Z_t$) from the experience bank using cosine similarity. No external retriever is needed.
   • Cross-Attention Aggregation: The retrieved embeddings are aggregated via a cross-attention mechanism using a learnable query token ($u$). This allows the model to dynamically weight experiences based on their relevance to the current generation state.
   • Gated Integration: The aggregated experience signal is integrated into the hidden state through a gated residual mechanism. Learnable retention ($r_t$) and input ($i_t$) gates, along with a mixing coefficient ($a_t$), control the balance between preserving the original hidden state and incorporating external knowledge. This prevents excessive experience influence from disrupting the model's reasoning flow.

3. End-to-End Training with Reinforcement Learning:

   • The entire pipeline (policy $\theta$ and experience parameters $\psi_{exp}$) is optimized jointly using Group Relative Policy Optimization (GRPO).
   • The framework supports both generative tasks (e.g., QA, coding) by producing textual outputs via experience-enhanced decoding, and ranking tasks (e.g., recommendation) by computing relevance scores between the final hidden state and candidate embeddings.

Key Contributions

• Unified Latent Space Framework: ExpWeaver unifies retrieval and generation within a shared representational space, eliminating the need for a separate retriever and avoiding the token overhead of text concatenation.
• Adaptive Retrieval: By performing retrieval at every decoding step based on the current hidden state, the system learns query-dependent experience utilization that captures deeper structural patterns rather than superficial text overlap.
• Task Agnosticism: The architecture naturally extends to diverse task types, including question answering, mathematical reasoning, code generation, scientific prediction, and sequential recommendation.

Experimental Results
The authors evaluated ExpWeaver on ExpBench, a benchmark spanning 13 diverse tasks across three scenarios: Generic (QA, Reasoning, Coding), Scientific (Pharmaceutical Chemistry), and Recommendation.

• Performance: ExpWeaver achieved state-of-the-art results on 12 out of 13 tasks, outperforming the strongest baseline (Search-R1) by an average of 6.8%.
• Token Efficiency: While text-based retrieval methods like Search-R1 required 1.5–2× more tokens due to explicit concatenation, ExpWeaver maintained token efficiency comparable to non-retrieval baselines (e.g., HRPO), consuming only ~10% more tokens while delivering significantly higher accuracy.
• Generalization: ExpWeaver demonstrated superior cross-domain generalization. In zero-shot transfer settings, it outperformed the strongest baseline by 16.32%, and by 15.21% in few-shot transfer, indicating that latent experience representations capture more transferable problem-solving patterns.
• Ablation Studies: Experiments confirmed the necessity of cross-attention aggregation over simple pooling and identified an optimal initialization range for the mixing coefficient to balance experience integration with hidden state preservation.

Significance and Claims
The paper claims that ExpWeaver represents a significant shift in experience learning for LLM agents. By moving from explicit text-level operations to latent space integration, the framework overcomes the scalability and adaptability bottlenecks of previous methods. The authors assert that this approach enables agents to learn more efficiently from past interactions, achieving superior performance with lower computational costs. Furthermore, the demonstrated cross-domain generalization suggests that latent experience representations are robust enough to transfer learned patterns to unseen domains, potentially accelerating development in specialized fields like scientific research where labeled data is scarce. The work positions ExpWeaver as a step toward more efficient, unified, and effective knowledge utilization in large language models.