Technical Summary: Evidence Over Plans: Online Trajectory Verification for Skill Distillation

Problem Statement

While Large Language Model (LLM) agents can improve task success rates using human-written procedural skills, the quality of these skills is difficult to assess without environment-grounded verification. Existing skill generation methods often rely on preference logs or prior plans rather than direct interaction with the environment. This leads to a fundamental bottleneck: skills are frequently "prior-based," meaning they encode generic heuristics or the model's initial intent rather than empirical evidence discovered through execution. Consequently, these methods often yield negligible or even degraded gains when transferring skills between models of different capabilities, as they lack the specific constraints, dependencies, and failure modes that only emerge during actual task execution.

Methodology: SPARK and PDI

The authors propose SPARK (Structured Pipelines for Autonomous Runnable tasKs and sKill generation), a framework designed to generate skills grounded in environment-verified evidence and to analyze the quality of the trajectories from which they are distilled.

1. The SPARK Pipeline

SPARK operates through two interconnected pipelines:

• Skill Generation Pipeline: A "teacher" agent interacts with a Dockerized environment for up to $N_{max}$ attempts.
  • Exploration Memo: Instead of carrying raw logs, the agent maintains a structured "exploration memo" that summarizes attempts, commands, verified facts, current error patterns, and next strategies. This memo is rewritten (not appended) after each failure to prevent context dilution.
  • Evidence Assembly: Upon success, the system assembles six structured evidence blocks: Task Pattern, Execution Chain, Verification records, Lessons (failure patterns), Environment context, and Raw Support Tail.
  • Distillation: These blocks are distilled into a reusable SKILL.md document.
• Task Construction Pipeline: To ensure rigorous evaluation, SPARK converts natural language prompts into structured task blueprints. These undergo iterative critique, repair, and deterministic oracle verification (using pytest) to create self-contained, runnable benchmark instances.

2. The Posterior Distillation Index (PDI)

To quantify whether a skill is grounded in posterior evidence (environment interaction) rather than prior plans, the authors introduce the Posterior Distillation Index (PDI). PDI is a trajectory-level metric computed as an equal-weight linear combination of three z-scored signals:

1. Execution Grounding ($\phi_{exec}$): Measures the distributional similarity between the agent's successful execution commands and the final skill. High values indicate the skill focuses on operations verified by the environment.
2. Plan Copying ($\phi_{plan}$): Measures the similarity between the "Next Strategy" sections of the exploration memo and the final skill. Low values are desirable, indicating the skill diverges from the agent's initial plans and incorporates new evidence.
3. Memo Ossification ($\phi_{oss}$): Measures the stability of "Verified Facts" and failed test sets across attempts. Low values are desirable, indicating the agent's understanding evolved rather than repeating the same failure patterns (cognitive stagnation).

$$PDI = z(\phi_{exec}) - z(\phi_{plan}) - z(\phi_{oss})$$

3. Online Intervention

A key innovation is using a proxy of PDI as an online diagnostic signal. During skill exploration, if the proxy-PDI drops below a threshold, SPARK triggers interventions:

• Soft Intervention: Encourages broader hypothesis revision.
• Strong Intervention: Withholds the previous strategy and forces the agent to anchor on verified facts and error patterns, preventing the agent from persisting with a stale plan.

Key Contributions

1. Trajectory-Verified Metric (PDI): A principled, interpretable metric that distinguishes skills grounded in environment evidence from those based on unverified prior plans. It provides a mechanistic assessment of which trajectories yield transferable skills.
2. Transparent Skill Generation (SPARK): A framework that preserves execution logs, verifier signals, and memo histories, enabling full trajectory-level analysis and making the distillation process analyzable.
3. Online Intervention Mechanism: A method to use PDI proxies to monitor and intervene during the skill exploration process, actively improving the quality of generated skills before distillation is complete.
4. Trajectory Benchmark: A dataset of 86 runnable tasks (extended to 300 variants) with full execution traces, verifier signals, and distilled skills, allowing for the evaluation of skill transfer across environment interactions.

Experimental Results

The authors evaluated SPARK on 86 tasks from the SkillsBench benchmark and 300 generated task variants, as well as the ALFWorld environment.

• Performance Gains: SPARK-generated skills consistently outperformed both "no-skill" baselines and human-written skills across seven student models (ranging from GPT-5.4-nano to DeepSeek-Chat). Notably, small student models equipped with SPARK skills often surpassed the unaided performance of the strong teacher models used to generate the skills.
• Cost Efficiency: While teacher exploration is expensive, the distilled skills allow weaker student models to solve tasks at an inference cost up to 1,000× cheaper than the teacher's exploration cost.
• PDI Predictive Power:
  • There is a significant positive correlation ($\rho \approx 0.36$) between PDI and skill gain.
  • Skills from high-PDI trajectories achieved the highest pass-gain rates.
  • "Plan copying" (high $\phi_{plan}$) combined with low execution grounding resulted in near-zero or negative gains, confirming that summarizing plans without evidence is ineffective.
  • "Memo ossification" (high $\phi_{oss}$) correlated negatively with skill quality, indicating that cognitive stagnation harms transfer.
• Online Intervention: Ablation studies showed that enabling PDI-guided online intervention significantly improved downstream pass rates (e.g., increasing DeepSeek-Chat's pass rate from 9.1% to 42.4% on specific subsets).
• Generalization: SPARK skills generalized across task variants and even to the out-of-distribution ALFWorld environment, improving success rates from 16.7% to 40.0%.

Significance and Claims

The paper argues that effective procedural knowledge for agents must be posterior-based, meaning it must encode constraints and failure modes discoverable only through environmental interaction. The authors claim that:

• Timing is Critical: Robust skills should be distilled from empirical interaction rather than prior plans.
• Evidence Over Plans: The quality of a skill is determined by its grounding in verified evidence, not the complexity of the agent's initial plan.
• Transferability: By using PDI to filter and guide distillation, SPARK produces skills that are efficient, transferable, and capable of enabling weaker models to outperform stronger models that lack these specific, environment-grounded insights.
• Mechanistic Understanding: The framework moves beyond black-box skill generation, providing a mechanistic understanding of why some trajectories yield useful skills (divergent exploration, low ossification) while others do not.

The authors conclude that autonomous skill generation works most effectively when the agent learns from the environment as a prerequisite, not an afterthought, for producing vital knowledge that future agents can inherit and execute verifiably.