Technical Summary: LifeSkill – Skill-Enhanced Test-Time Co-Evolution for Online Lifelong Learning Agents

1. Problem Statement

Large Language Model (LLM) agents operating in dynamic, interactive environments require lifelong learning capabilities to adapt to evolving task distributions and accumulate knowledge from experience. However, existing approaches suffer from three critical limitations:

1. Static Parameters: Most agents keep model parameters fixed during deployment, relying solely on external memory retrieval (e.g., Synapse, AWM) to improve performance. This prevents true learning during the interaction phase.
2. Insufficient Skill Supervision: While agents can summarize trajectories or reflect on failures, extracting reusable skills from interaction feedback lacks reliable, execution-grounded supervision. Current methods often rely on subjective LLM judges or heuristic summaries rather than task success.
3. External Knowledge Representation: Acquired knowledge is typically stored as discrete external text. This prevents the model from internalizing, composing, and generalizing these skills as part of its parametric knowledge, leading to context bloat and an inability to transform experience into lasting capability.

The core challenge is to enable agents to continuously improve while acting in the environment, converting test-time feedback into persistent parametric growth without relying on static inference or external memory retrieval alone.

2. Methodology: LifeSkill Framework

The authors propose LifeSkill, a two-stage reinforcement learning framework designed for Online Lifelong Learning Agents. The framework operates on a stream of tasks ($D_{stream}$) and maintains two models with the same backbone but distinct parameters:

• Policy Model ($\pi_\theta$): Executes tasks.
• Skill Extraction Model ($\pi_\phi$): Distills reusable skills from failed interactions.

These models engage in a test-time co-evolution loop consisting of two key mechanisms:

A. Verifier-Guided Skill Learning (VGSL)

This stage addresses the lack of direct supervision for skill extraction.

• Process: When the Policy Model fails a task ($r=0$), the Skill Extraction Model generates candidate skills ($s$) based on the failure trajectory.
• Evaluation: Instead of using a subjective LLM judge, each candidate skill is evaluated by the average verifier success of multiple skill-conditioned policy rollouts ($K_\tau$ trajectories per skill).
• Optimization: The Skill Extraction Model is trained using the DAPO (Direct Preference Optimization) objective. The utility of a skill is defined by its empirical success rate ($R_i$). Skills that lead to higher verification success rates are rewarded, training the extractor to generate skills that are execution-grounded rather than merely textually plausible.

B. Online Skill Internalization (OSI)

This stage addresses the mismatch between skill-guided exploration and the need for parametric adaptation.

• Process: For successful trajectories where a skill guided the agent to a solution ($r=1$), the explicit skill text is removed from the input context.
• Optimization: The Policy Model is updated using the verifier reward associated with the trajectory, conditioned only on the original task input ($q$), not the skill ($s$).
• Goal: This forces the model to internalize the reasoning capabilities and behaviors discovered during the skill-guided exploration directly into its parameters. It transforms successful "scaffolded" behaviors into "unscaffolded" parametric knowledge, eliminating the need to regenerate or retrieve the skill text in future interactions.

3. Key Contributions

1. LifeSkill Framework: A novel two-stage reinforcement learning framework that enables online lifelong LLM agents to improve during deployment, moving beyond static-parameter inference with external memory.
2. Verifier-Guided Skill Learning: A method to train a skill extractor using execution-grounded feedback (verifier success rates) rather than subjective scoring, ensuring extracted skills are practically useful for task solving.
3. Online Skill Internalization: A mechanism to convert successful skill-guided trajectories into unscaffolded policy improvements, allowing the agent to absorb reasoning capabilities into its parameters and reduce dependence on retrieved text.
4. Empirical Validation: Extensive experiments demonstrating that coupling execution-grounded skill extraction with online parametric adaptation yields superior performance compared to existing baselines.

4. Experimental Results

The framework was evaluated on LifelongAgentBench, a benchmark for continual adaptation in long-horizon interactive tasks across three environments: Database (DB), Operating System (OS), and Knowledge Graph (KG).

• Performance: LifeSkill achieved an average accuracy of 0.59, outperforming the strongest training-free baseline (Group Self-Consistency) by 10 absolute points and the strongest training-based baseline (RFT) by 7 absolute points.
• Environment Specifics:
  • DB: LifeSkill reached 0.82 (vs. 0.63 for the best baseline).
  • OS: LifeSkill reached 0.64 (vs. 0.56 for the best baseline).
  • KG: LifeSkill scored 0.32, slightly below the best training-free baseline (0.36). The authors attribute this to the sparsity of binary reward signals in long-horizon KG tasks, which weakens the learning signal for both skill extraction and internalization.
• Ablation Studies:
  • Replacing VGSL with LLM-Judge supervision reduced performance, confirming the necessity of execution-grounded rewards.
  • Removing the Skill Extraction Model caused the largest performance drop, highlighting the value of dual-model co-evolution.
  • Disabling OSI reduced performance, proving that skill extraction alone is insufficient without internalization into parameters.
• Hyperparameters: Performance improved with increased test-time compute (more rollouts) up to a saturation point (4 rollouts), and optimal window sizes varied by environment (DB favored smaller windows for faster updates; OS favored larger windows for stability).

5. Significance and Claims

The paper claims that LifeSkill represents a significant step toward genuine continual adaptation for LLM agents. By treating skills as an intermediate learning interface that guides exploration and is subsequently internalized, the framework bridges the gap between experience retrieval and parametric learning.

The authors emphasize that their approach allows agents to:

• Learn which skills to extract from failures using reliable, task-specific verifiers.
• Perform online parameter updates during deployment, rather than relying solely on external memory.
• Internalize successful behaviors under the original task input, reducing long-term dependence on retrieved text and mitigating context bloat.

The work suggests that while current agents can retrieve experience, LifeSkill demonstrates a pathway to effectively transform that experience into lasting, internalized capability. Future work is identified as improving performance in sparse-reward, long-horizon settings and reducing the test-time compute costs associated with skill-guided exploration.