💬 NLP

Ask Only When Needed: Proactive Retrieval from Memory and Skills for Experience-Driven Lifelong Agents

This paper introduces ProactAgent, a lifelong learning framework that employs Experience-Enhanced Online Evolution and a Proactive Reinforcement Learning-based Retrieval mechanism to enable agents to dynamically and selectively retrieve structured memories and skills only when necessary, thereby significantly improving task performance and efficiency across diverse environments.

Original authors: Yuxuan Cai, Jie Zhou, Qin Chen, Liang He

Published 2026-04-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuxuan Cai, Jie Zhou, Qin Chen, Liang He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot butler to manage your entire household. You want it to learn from every mistake and success so it gets better over time.

The Problem with Old Robots:
Most current "lifelong learning" robots are a bit like students who only open their textbooks at the very start of an exam.

Static Start: They read a few notes before the task begins, then try to solve everything from memory. If they forget something halfway through, they can't look it up.
Constant Chatter: Other robots try to look up everything at every single step, like someone who asks a librarian for a book before every sentence they speak. This is slow, annoying, and fills their brain with too much noise.
The "Gated" Robot: Some use a separate, expensive manager to decide, "Hey, do you need to look something up?" This adds cost and delay.

The Solution: PROACTAGENT
The authors of this paper built a new kind of robot called PROACTAGENT. Think of it as a super-intelligent apprentice who knows exactly when to ask for help and exactly what to ask.

Here is how it works, using simple analogies:

1. The Organized Library (Structured Experience Base)

Instead of a messy pile of notes, this robot has a perfectly organized library with five specific shelves:

Facts Shelf: Hard truths (e.g., "Aluminum foil conducts electricity").
Story Shelf: Past episodes (e.g., "Last time I tried to fix the sink, I broke the pipe").
Success Skills: Recipes for winning (e.g., "Always check the battery before turning on the device").
Failure Skills: Warnings of what not to do (e.g., "Never mix these two chemicals").
Comparison Skills: The "Why" (e.g., "Method A worked better than Method B because...").

This means when the robot needs help, it doesn't just get a random page; it gets the right kind of help for the situation.

2. The "What If?" Game (Proactive Retrieval)

This is the magic trick. The robot doesn't just guess when to look things up. It plays a mental game of "What If?"

Imagine the robot is about to make a move. It pauses and thinks:

Scenario A: "If I look up the answer right now, what happens?"
Scenario B: "If I don't look it up and just guess, what happens?"

It simulates both paths in its head.

If Scenario A (looking it up) leads to a happy ending and Scenario B leads to a crash, the robot learns: "Aha! Looking it up was necessary. I should do that next time."
If Scenario A and Scenario B lead to the same result, the robot learns: "Looking it up was a waste of time. I'll save my energy next time."

This teaches the robot to be proactive. It learns to feel a "knowledge gap" and fill it only when it truly matters.

3. The Double-Edged Sword (Joint Evolution)

Most robots either update their "memory" (the library) OR update their "brain" (how they think), but not both at the same time.

PROACTAGENT does both simultaneously.
As it solves tasks, it writes better notes into the library.
As it reads better notes, it becomes smarter at solving tasks.
It's a self-reinforcing loop: The better it gets, the better its notes become, which makes it even better.

The Results

When they tested this robot on three different challenges (a science quiz, a house-cleaning simulation, and a student life manager):

It won more often: It solved tasks much more successfully than robots that just guessed or looked things up randomly.
It was faster: Because it stopped asking unnecessary questions, it finished tasks in fewer steps.
It was efficient: A smaller version of this robot (3 billion "brain cells") performed almost as well as a massive, expensive robot (7 billion "brain cells") that didn't have this smart retrieval system.

The Takeaway

The paper proves that for an AI to truly be "lifelong," it shouldn't just be a giant database or a super-smart guesser. It needs to be a strategic learner that knows exactly when to consult its past experiences and what to ask, turning every interaction into a lesson that makes it smarter for the next one.

1. Problem Statement

Current online lifelong learning agents face two critical limitations in utilizing past experience:

Passive Retrieval: Existing methods trigger memory retrieval based on static rules (e.g., only at task start), continuous heuristics (every step), or external gating models. Agents lack the intrinsic capability to recognize knowledge gaps during interaction and proactively decide when and what to retrieve. This leads to context overload, redundancy, or missed opportunities for guidance.
Decoupled Evolution: Most systems treat textual memory accumulation (storing facts/episodes) and policy optimization (updating the agent's decision-making parameters) as independent processes. Updating only one limits the agent's ability to continually improve, as a better policy generates better data for memory, and better memory should guide a better policy.

The core challenge is enabling an agent to proactively identify when its current knowledge is insufficient, retrieve the most relevant structured experience, and jointly evolve its memory and policy to improve future performance.

2. Methodology: PROACTAGENT

The authors propose PROACTAGENT, a framework that unifies proactive retrieval with experience-enhanced online evolution over a structured experience base. It consists of two tightly coupled components:

A. Experience-Enhanced Online Evolution (EXPONEVO)

This component creates a closed loop where acting, experience accumulation, and policy optimization reinforce each other.

Structured Experience Base: Instead of a monolithic memory, the experience base ( $D$ $D$ ) is organized into five typed repositories:
- Memory ( $M$ ): Factual memory ( $M_f$ ) and Episodic memory ( $M_e$ ).
- Skills ( $S$ ): Success skills ( $S_+$ ), Failure skills ( $S_-$ ), and Comparative skills ( $S_\Delta$ ).
- Benefit: A single query returns complementary evidence (facts) and actionable guidance (skills), reducing redundancy.
Joint Evolution: The agent updates both the external memory (distilling new entries from trajectories) and the internal policy ( $\pi_\theta$ ) simultaneously.

B. Proactive Reinforcement Learning-based Retrieval (PROACTRL)

This component formulates retrieval as an explicit policy action ( $A_{ret}$ ) and learns the optimal timing and content of retrieval via Paired-Branch Process Rewards.

Paired-Branch Rollout: When a retrieval action is triggered at step $t_b$ $t_{b}$ , the system adaptively samples two continuations from the identical interaction prefix:
1. Retrieval Branch ( $\tau_{ret}$ ): Proceeds with the retrieved context.
2. No-Retrieval Branch ( $\tau_{no-ret}$ ): Suppresses the retrieval action and continues with the original policy.
Process Reward ( $r_{proc}$ ): The system compares the outcomes of the two branches.
- If $\tau_{ret}$ outperforms $\tau_{no-ret}$ , a positive reward is assigned, reinforcing that retrieval was necessary.
- If $\tau_{ret}$ underperforms, a negative reward is assigned, teaching the agent that the retrieval was redundant or harmful.
Efficiency Penalty: Additional penalties are applied for repeating queries or taking longer paths than necessary.
Optimization: The policy is updated using Group Relative Policy Optimization (GRPO) with these step-level process rewards, allowing the agent to learn when to retrieve and what to query.

3. Key Contributions

Proactive Retrieval Framework: Introduced PROACTAGENT, the first framework to treat retrieval as a learned, intrinsic policy action rather than a passive heuristic, enabling agents to self-diagnose knowledge gaps.
Structured Experience Base: Designed a typed repository (Facts, Episodes, Success/Failure/Comparative Skills) that provides both evidence and behavioral guidance, preventing context overload.
Paired-Branch Process Rewards: Developed a novel RL mechanism that uses counterfactual rollouts (comparing retrieval vs. no-retrieval from the same prefix) to provide step-level supervision for retrieval decisions, solving the credit assignment problem for retrieval actions.
Joint Co-Evolution: Demonstrated that unifying memory refinement and policy optimization yields significantly better performance than evolving either in isolation.

4. Experimental Results

The framework was evaluated on three interactive benchmarks: SciWorld, AlfWorld, and StuLife.

Performance Gains:
- SciWorld: Achieved 73.50% success rate (SR), an 18-point improvement over the strong baseline GRPO+Reflexion (55.50%).
- AlfWorld: Achieved 71.28% SR, outperforming all online baselines.
- StuLife: Achieved performance competitive with proprietary models, demonstrating effectiveness in long-horizon, cross-task dependency scenarios.
Efficiency:
- Reduced interaction rounds by 33.2% on SciWorld and 22.5% on AlfWorld compared to baselines.
- Significantly reduced token consumption (e.g., 0.43k tokens vs. 0.95k for baselines on SciWorld).
Model Scaling: A 3B-parameter model equipped with PROACTAGENT nearly matched the performance of a 7B-parameter baseline (GRPO+Reflexion), suggesting proactive retrieval can partially offset capacity gaps.
Ablation Studies: Confirmed that removing either the structured experience base or the paired-branch reward mechanism caused significant performance drops, validating the necessity of both components.

5. Significance

This work represents a paradigm shift in lifelong learning for agents:

From Passive to Active: It moves retrieval from a static or rule-based mechanism to a learned, adaptive capability where the agent decides if and what to retrieve based on immediate needs.
Efficiency vs. Capacity: It demonstrates that for long-horizon tasks, learning to control retrieval is more critical than simply scaling model parameters. Proactive agents achieve higher success with fewer steps and less compute.
Self-Reinforcing Loop: By jointly evolving memory and policy, the system creates a virtuous cycle where better policies generate better data, which in turn refines the memory to guide future policies more effectively.

In conclusion, PROACTAGENT establishes that effective lifelong agents must not only accumulate experience but also learn to proactively retrieve and jointly evolve their knowledge and decision-making strategies to operate efficiently in complex, dynamic environments.