Technical Summary: Harnessing LLM Agents with Skill Programs (HASP)

1. Problem Statement

While Large Language Model (LLM) agents have demonstrated increasing capability in planning and interacting with environments, they frequently exhibit recurring failure patterns, such as terminating before verification, committing to brittle intermediate conclusions, or repeating unproductive actions. Existing approaches to mitigate these failures often rely on textual skills derived from past experience. These skills are typically injected into prompts or used as advisory guidance. However, the paper argues that textual skills suffer from three core limitations:

1. Advisory Nature: They express what an agent should do in principle but lack explicit mechanisms for when to activate or how to intervene within the policy loop.
2. Weak Control: They are often ignored by the model in practice, failing to reliably alter agent behavior.
3. Lack of Executability: There is a gap between reusable experience expressed in language and reusable experience that can explicitly control agent actions.

The central challenge is to bridge this gap by transforming passive, textual lessons into active, executable mechanisms that can intervene directly in the agent's decision-making process.

2. Methodology: The HASP Framework

The authors introduce HASP (Harnessing LLM Agents with Skill Programs), a framework that upgrades skills from passive text to Program Functions (PFs). A PF is defined as a reusable, executable state-action intervention function.

Core Components

• Program Functions (PFs): Unlike natural language reminders, PFs are executable Python modules with two distinct interfaces:
  • should_activate(state, action): A deterministic predicate that decides if the skill is relevant to the current state and proposed action.
  • intervene(state, action, teacher): Returns a structured intervention, which can be:
    • Modify_Action: Rewriting or refining the next action (e.g., shortening an over-constrained search query).
    • Inject_Context: Appending corrective text to the agent's observation (e.g., a warning about entity confusion).
    • NOOP: Abstaining from change while logging an audit record.
• Skill Library: A collection of validated PFs. New candidates are generated from failure cases, undergo syntax and interface validation, and are reviewed by a teacher model before admission.
• The Harness: An external control layer wrapping the base agent policy ($\pi_\theta$). At each step, the harness retrieves relevant PFs, evaluates their activation predicates, executes valid interventions, and feeds the revised action or injected context back into the loop.

Three Operational Modes

HASP is designed to be highly modular, supporting three distinct paradigms:

1. Inference-Time Intervention: PFs act as executable guardrails at runtime. They trigger autonomously based on the agent's state to modify actions or inject context without updating the model weights. An optional auxiliary teacher can assist in selecting the most appropriate PF when multiple candidates exist.
2. Post-Training (Internalization): PF executions generate structured records containing the original action, the PF repair, and the outcome. These records are scored using four signals: Timing (when), Mode (how), Correctness (quality), and Outcome (effectiveness). These scores guide post-training via:
   • Supervised Fine-Tuning (SFT): Training on corrected actions.
   • Rejection Sampling (RS): Selecting trajectories that match PF-guided corrections and achieve task success.
   • On-Policy Distillation (OPD): Training the student on its own states where PFs intervened.
3. Self-Improving Evolution: The system revisits residual failures under the current checkpoint, summarizes recurring failure-repair patterns into candidate PFs, and updates the external skill library after strict executable validation and teacher review. This closes the loop between execution, learning, and library growth.

3. Key Contributions

The paper outlines three primary contributions:

1. Skills as Executable State-Action Functions: The transformation of reusable agent experience into executable Program Functions that trigger on demand and intervene explicitly, moving beyond passive textual skills.
2. A Modular Agent Harness Framework: The proposal of HASP, which supports controllable PF triggering and intervention across inference-only, post-training, and self-improving paradigms.
3. Strong Empirical Performance: Demonstrating substantial gains over competitive baselines across web-search reasoning, mathematical reasoning, and coding tasks.

4. Experimental Results

The authors evaluated HASP on HotpotQA, 2Wiki, and MuSiQue (web-search), AIME24, AMC23, and GameOf24 (math), and HumanEval, MBPP, and BigCodeBench (coding). The backbone model was Qwen2.5-7B-Instruct.

Key Findings

• Inference-Time Gains: Even without model updates, PF-only intervention significantly outperformed baselines.
  • On web-search reasoning, PF-only intervention improved average accuracy to 51.0% (vs. 20.5% for Prompt-Only Skills and 31.2% for the multi-loop ReAct baseline).
  • Adding an auxiliary teacher for PF selection further boosted web-search accuracy to 56.2%.
  • On coding, PF-only reached 63.4% pass@1, rising to 68.7% with teacher selection.
• Post-Training Internalization: PF-derived supervision allowed the model to internalize behaviors without full reinforcement learning.
  • On web-search, On-Policy Distillation (OPD) achieved 62.5%, and Rejection Sampling (RS) reached 59.3% under a fixed library.
  • These results were competitive with or superior to standard SFT and skill-based training schemes.
• Closed-Loop Evolution: Controlled evolution of the skill library further improved performance.
  • HASP-Evolve + RS achieved 60.3% on web-search and 45.4% on mathematical reasoning.
  • On coding, it reached 69.9% pass@1, outperforming vanilla SFT by 12.4% and remaining competitive with GRPO-based methods.
• Mechanism Analysis:
  • Intervention Types: 65.1% of interventions were action-level modifications (e.g., rewriting queries), while 34.9% were context injections.
  • Signal Importance: Ablation studies showed that all four supervision signals (Timing, Mode, Correctness, Outcome) were necessary. Removing "Mode" caused the largest drop (-15.5%), indicating that how an intervention is applied is as critical as when.
  • Filtering Necessity: Strict filtering (executable validation + teacher review) was essential. Unfiltered evolution caused a sharp performance drop (-24.0%), confirming that library pollution degrades agent performance.

5. Significance and Claims

The paper positions HASP as a complementary path for improving LLM agents. The authors claim that:

• Stabilization over Discovery: PFs help surface and stabilize useful strategies that the model can often execute but does not apply reliably. This contrasts with RL-style exploration, which is better suited for discovering fundamentally new strategies.
• Explicit Control: By converting skills into executable code, HASP provides explicit, auditable control over agent behavior, addressing the "advisory" weakness of prompt-based skills.
• Modularity: The framework decouples the skill definition from the training objective, allowing the same PFs to be used for runtime intervention, supervised training, or self-improvement.

The authors remain modest regarding limitations, noting that PFs primarily help absorb existing strategies rather than discover new ones, that the framework relies on clear verification signals (limiting applicability to open-ended tasks), and that self-improvement requires strict filtering to prevent library degradation. They conclude that while broader RL-style exploration may still be needed for novel strategy discovery, PFs offer a robust mechanism for refining and stabilizing agent behavior in complex, long-horizon tasks.