🤖 AI

PATH-Bench: Path-Dependent Evaluation of Lifelong Agents

This paper introduces PATH-Bench, a benchmark for evaluating how the sequence of accumulated experiences impacts lifelong LLM agents, and proposes Selective Experience Use (SEU), a method that filters path-dependent memories to reduce forgetting and improve forward transfer.

Original authors: Xidong Yang, Xingyi Zhang, Wenhao Li, Wenyan Liu, Junjie Sheng, Yun Hua, Wei Yin, Tao Fang, Chuyun Shen, Xiangfeng Wang

Published 2026-08-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xidong Yang, Xingyi Zhang, Wenhao Li, Wenyan Liu, Junjie Sheng, Yun Hua, Wei Yin, Tao Fang, Chuyun Shen, Xiangfeng Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a super-smart robot to be a helpful assistant. In the world of artificial intelligence, there's a big dream: creating "lifelong" agents that don't just answer questions once and forget, but actually learn from their past interactions, build a memory, and get better over time. Think of it like a student who doesn't just study for a test and then toss the notes away, but keeps a growing notebook of tricks, facts, and lessons learned from every single day.

However, there's a tricky problem. If you just dump every single thing a robot learns into its memory, it might get confused. Imagine a student trying to solve a math problem while their brain is flooded with memories of a soccer game, a cooking recipe, and a history lesson all at once. Some of those memories might help, but others might get in the way. Scientists call this "path dependence"—it means that how and in what order the robot learns things changes what it remembers and how well it performs later. The big question is: Does the robot's path of learning matter, and if so, how do we make sure it keeps the good stuff and forgets the bad?

This is exactly what the researchers behind PATH-Bench set out to investigate. They built a special testing ground to see how different AI agents handle their growing memories. Instead of just watching them learn randomly, they created a controlled experiment where they could feed the agents specific "histories"—some full of helpful tips, others full of confusing or misleading advice. They wanted to see if the order of these experiences changed the agents' ability to solve new tasks, and whether the agents could tell the difference between a helpful memory and a distracting one.

Here is what they discovered, and it's a bit more complicated than just "more memory is better."

First, they found that having a memory doesn't automatically make an AI smarter. In fact, for some agents, adding a memory bank actually made them perform worse than if they had no memory at all! It turns out that just storing past interactions isn't enough; the agent needs to know which memories are useful for the current job. If an agent grabs a memory that sounds similar but is actually irrelevant, it can get confused and make mistakes. It's like trying to fix a leaky faucet by reading a manual on how to bake a cake; the book is real, but it's the wrong tool for the job.

Second, the researchers discovered that what works as a memory depends entirely on the type of task. For simple, one-step tasks (like writing a single piece of code), agents that remembered specific, concrete details from the past did best. But for complex, multi-step tasks (like using a bunch of different tools to plan a trip), agents that remembered high-level patterns and abstract rules performed much better. It's the difference between remembering the exact ingredients you used for one specific cake versus remembering the general rule that "eggs make things fluffy." One approach wins for the cake, the other wins for the general baking.

Third, and perhaps most surprisingly, they found that learning something new doesn't mean you keep it forever. An agent might show a huge improvement right after learning a new skill (called "forward transfer"), but then completely forget that skill later on when faced with new, confusing experiences. Conversely, a new experience can sometimes reshape or even erase gains the agent made earlier. It's like learning a dance move, getting really good at it, and then having your brain get so cluttered with new, conflicting dance moves that you forget the original one entirely. The path you take matters: if you learn the "right" sequence of things, you keep your skills; if you learn the "wrong" sequence, you might lose them.

To fix these issues, the authors proposed a new strategy called Selective Experience Use (SEU). Think of this as a smart filter or a bouncer for the agent's memory. Instead of letting every single retrieved memory into the agent's brain, SEU checks each one against the current task. If a memory is directly helpful, it gets let in. If it's only useful as a general pattern, it gets simplified and let in. But if it's likely to cause confusion or interference, it gets blocked.

When they tested this new filter, the results were promising. Across almost all the different agents and tasks they tried, SEU consistently reduced the amount of "forgetting" and helped the agents transfer their skills to new problems more effectively. It didn't just make them remember more; it made them remember better.

In short, this paper teaches us that for AI to truly learn like a lifelong student, it can't just hoard every experience. It needs to be picky. It needs to understand the shape of the task it's facing and curate its memories carefully, letting in the helpful lessons and filtering out the noise. The path of learning isn't just a background detail; it's the main character in the story of how an AI grows up.

Technical Summary: PATH-Bench: Path-Dependent Evaluation of Lifelong Agents

Problem Statement

Lifelong Large Language Model (LLM) agents aim to accumulate and preserve useful experience over time, typically by maintaining an external learning state (memories or skills) while keeping the backbone model frozen to avoid catastrophic forgetting and high computational costs. However, existing benchmarks for lifelong learning largely overlook path dependence: the phenomenon where the specific sequence and history of prior tasks shape what an agent transfers and retains.

Current evaluations often rely on fixed or randomized task sequences, failing to control for how accumulated experience—whether helpful, misleading, or conflicting—interacts with subsequent tasks. Preliminary studies on LifelongAgentBench indicate that different task paths over the same task set can alter the relative ranking of agents. Furthermore, standard lifelong learning metrics (forward transfer, backward transfer, forgetting) are computationally expensive to apply at scale to LLM agents, which require multi-step reasoning and environment interaction. There is a critical need for a framework that efficiently traces how task path dependence shapes the lifelong performance of agents, distinguishing between the utility of experience representation and the structure of the learning path.

Methodology: PATH-Bench

The authors introduce PATH-Bench, a benchmark designed to evaluate lifelong agents under controlled, path-dependent histories. The framework operates through four main stages:

1. Task Pool Construction

The benchmark utilizes two distinct datasets to cover different interaction structures:

BigCodeBench: A single-turn code generation benchmark.
WildToolBench: A multi-turn tool-use benchmark grounded in real-world user behavior.
Tasks are filtered based on "headroom" (ensuring they are neither too easy nor too hard for the base LLM) to guarantee measurable improvement potential.

2. In-Context Transfer Estimation

To avoid the computational cost of simulating full agent evolution for every task pair, PATH-Bench estimates directed task relationships using multi-model in-context learning (ICL).

For every ordered pair of tasks $(t_i, t_j)$ , a frozen LLM is prompted with $t_i$ as a one-shot demonstration to solve $t_j$ .
The transfer gain $M_{i,j}$ is calculated as the performance difference between the demonstration-conditioned and zero-shot performance.
Three models (DeepSeek-V4-Flash, GPT-5.4-mini, GLM-5.0) vote on the sign of the gain (Positive, Neutral, Negative). A consensus (at least two models) determines the final relationship matrix $V$ , excluding pairs with mixed votes.

3. Probe-Centered Sequence Sampling

Instead of evaluating full task sets repeatedly, PATH-Bench constructs sequences centered around a specific probe task ( $T_{probe}$ ).

History Construction: The sampler selects a probe task and constructs an intervening history of tasks based on the transfer matrix $V$ . Histories are configured as positive-dominant (mostly helpful tasks) or negative-dominant (mostly interfering tasks).
Protocol: The sequence begins with a cold-start probe, followed by a warm-up block, and then interleaved intervening tasks with the probe reappearing at fixed intervals. Only intervening tasks update the agent's external state; probe reappearances serve solely as evaluation points.

4. Evaluation Metrics

The framework quantifies performance using four metrics derived from the probe trajectory:

Average Performance (AP): Overall task-solving ability across the entire sequence.
Forward Transfer (FWT): Improvement of later probe occurrences relative to the cold-start baseline.
Backward Transfer (BWT): The beneficial retroactive effect of later experience on the probe (how much subsequent probes rise above the warm-up reference).
Forgetting (FGT): The degradation of probe capability relative to the post-warm-up reference.

Key Contributions

1. The PATH-Bench Framework

PATH-Bench is presented as the first benchmark to trace transfer and retention in LLM agents under controlled histories. It replaces exhaustive history enumeration with relationship-guided sampling and replaces full-set re-evaluation with longitudinal probe tracking.

2. Empirical Study of Eight Agents

The authors evaluated eight representative agents across three categories:

RAG Agents: BGE-M3 RAG, HippoRAG-v2.
Agentic Memory Agents: Clin, AWM, SimpleMem, MemRL.
Skill Agents: AutoSkill, SkillClaw.

Key Findings:

Experience Utility is Context-Dependent: Accumulated experience does not universally improve performance. Its utility depends on whether the representation of experience matches the task's interaction structure. For single-turn code tasks, task-specific concrete experience (e.g., AutoSkill) performed best. For multi-turn tool tasks, compact abstractions (e.g., Clin) outperformed concrete traces.
Dissociation of Transfer and Retention: Strong immediate forward transfer does not guarantee retention. Agents can exhibit high FWT while simultaneously suffering significant forgetting (FGT), particularly when the experience representation is mismatched to the task structure.
Path-Dependent Dynamics: Later experience can reshape gains acquired earlier. Even in positive-dominant histories, performance often erodes over time. Agents are highly susceptible to recency effects, where the most recent history interval dominates the agent's state, often overriding cumulative history.

3. Selective Experience Use (SEU)

Guided by the finding that indiscriminate injection of retrieved experience causes interference, the authors propose Selective Experience Use (SEU).

Mechanism: SEU acts as a harness between the retrieval module and the inference model. It evaluates retrieved items against the current task and decides to:
1. Preserve: Admit as concrete detail if directly reusable.
2. Abstract: Admit as compact guidance if only the high-level pattern transfers.
3. Ignore: Filter out potential interference.
Results: SEU was evaluated on a subset of 20 sequences drawn from the 50 sequences used in the main experiments. It consistently reduced forgetting across all agents, datasets, and transfer histories. It also improved forward transfer in the majority of settings, demonstrating that regulating the influence of path-accumulated experience is more critical than simply accumulating more data.

Significance and Claims

The paper claims that PATH-Bench provides a controlled evaluation framework that reveals limitations in current lifelong agent designs, specifically the lack of selectivity in experience usage. The authors argue that the bottleneck for lifelong agents is not the capacity to store experience, but the ability to judge its utility for the current task.

By demonstrating that experience utility depends jointly on representation and task structure, and that later experience can reshape earlier gains, the work offers actionable guidance for designing more robust agents. The proposed SEU harness serves as a proof-of-concept that filtering experience at inference time can decouple transfer from forgetting, leading to more stable lifelong learning.

The authors acknowledge limitations, noting that the current benchmark covers only two task domains (code generation and tool use) and utilizes a single backbone LLM, identifying the extension to broader domains and diverse models as a direction for future work.