Technical Summary: SKILLC – Learning Autonomous Skill Internalization in LLM Agents via Contrastive Credit Assignment

1. Problem Statement: Internalization Blindness

Structured skill prompts have become a practical mechanism for enhancing exploration and sample efficiency in long-horizon agentic Reinforcement Learning (RL). However, a dichotomy exists in current approaches:

• Skill-Augmented RL: Methods retain external skills at inference time, optimizing performance with runtime skill support.
• Skill-Internalization RL: Methods (e.g., SKILL0) withdraw skills during training to enable autonomous performance but suffer from a critical optimization mismatch termed "internalization blindness."

In existing internalization frameworks, policy updates (e.g., via GRPO) generate all rollouts for a task under a single prompt condition. Consequently, the gradient cannot distinguish between a successful trajectory that is truly autonomous and one that is merely skill-dependent. Even if a curriculum knows a skill is globally useful, the policy update reinforces success under the current regime rather than driving the agent toward success without the skill. This prevents the policy from learning to succeed autonomously during the training phase, leading to a persistent gap between "with-skill" and "without-skill" performance that only collapses (if at all) after a hard curriculum switch.

2. Methodology: The SKILLC Framework

The authors propose SKILLC, a framework based on Contrastive Skill Credit Assignment (CSCA). SKILLC converts the contrast between skill-injected and skill-free performance into a direct learning signal to resolve internalization blindness. The framework operates through three coordinated mechanisms:

A. Paired Contrastive Rollouts

For tasks within the active skill set, SKILLC samples paired rollouts within the same policy update:

• Skill-injected ($\tau^+$): Generated with a skill prompt ($z=1$).
• Skill-free ($\tau^-$): Generated without the skill prompt ($z=0$).
  Both use the same policy parameters $\pi_\theta$, creating a self-contrastive design. This allows the system to measure the task-level contrastive gap $\Delta(x) = E[R|x, z=1] - E[R|x, z=0]$. A positive gap indicates skill dependence; a gap near zero indicates internalization.

B. Dual-Stream Advantage Estimator

To inject this contrast into optimization without distorting global rankings, SKILLC employs a dual-stream advantage estimator:

1. Stream 1 (Global Ranking): Preserves the standard group-relative advantage using the mean and variance of all trajectories in the batch.
2. Stream 2 (Internalization Pressure): Applies a one-sided correction based on the contrastive gap.
   • For skill-injected trajectories ($z=1$), the advantage is downweighted by a correction term $C(x)$.
   • For skill-free trajectories ($z=0$), the advantage is upweighted by $C(x)$.
   • The correction magnitude $C(x)$ is proportional to the estimated gap $\hat{\Delta}(x)$ and an attribution strength $\lambda$.
     This mechanism explicitly redirects credit toward autonomous success ("succeed without skill support") while the agent is still learning, rather than waiting for a curriculum switch.

C. Internalization-Aware Curriculum

A smoothed validation-level signal drives an adaptive curriculum that controls three hyperparameters dynamically:

1. Attribution Strength ($\lambda$): Scales the contrastive correction.
2. Rollout Allocation ($\rho$): Adjusts the fraction of skill-injected vs. skill-free rollouts.
3. Active Skill Set Pruning: Monotonically retires skill types where the validation gap $\bar{\Delta}_{val}$ drops below a threshold.
   Unlike fixed-schedule methods, this curriculum is data-driven, responding to the actual measured progress of internalization.

3. Key Contributions

1. Identification of Internalization Blindness: The paper identifies that fixed-schedule internalization fails because standard policy gradients cannot distinguish autonomous success from skill-dependent success when all rollouts share a single prompt condition.
2. SKILLC Framework: The authors propose a solution utilizing paired contrastive rollouts and a dual-stream advantage estimator to make the internalization gap observable and actionable within every gradient step.
3. Adaptive Curriculum: A validation-driven mechanism that jointly modulates attribution strength, rollout budget, and skill retirement, ensuring the withdrawal of skills aligns with actual agent capability.
4. Critical Characterization of Applicability: The paper provides a nuanced analysis showing that CSCA's effectiveness is signal-quality-dependent. Strong contrastive signals yield substantial gains (e.g., +33.4% on ALFWorld's "Pick2" task), while weak signals on ceiling-effect tasks can cause performance regression.

4. Experimental Results

Experiments were conducted on ALFWorld (text-based household tasks) and WebShop (online shopping) using the Qwen2.5-7B-Instruct base model.

• Performance Gains: Without runtime skill access, SKILLC outperforms the strongest prior skill-internalization baseline (SKILL0) by 5.5% on ALFWorld (85.9% $\to$ 90.6%) and 4.4% on WebShop (70.9% $\to$ 74.0%).
• Competitiveness: SKILLC remains competitive with skill-augmented methods (which retain skills at inference). On ALFWorld, it matches the performance of D2Skill (a skill-augmented method) despite having no access to skills during inference.
• Ablation Studies:
  • Removing paired rollouts (single-condition updates) drops performance by 3.3%.
  • Replacing dual-stream normalization with mixed normalization drops performance by 2.2%.
  • Fixing the curriculum gate or removing pruning results in smaller but consistent performance drops.
• Training Dynamics: Unlike SKILL0, which maintains a large gap between with-skill and without-skill success rates until a hard switch, SKILLC demonstrates a monotonic decline in this gap, crossing zero during mid-training. This confirms the policy is learning autonomy continuously rather than relying on passive transfer.

5. Significance and Claims

The paper claims that SKILLC addresses a fundamental limitation in current agentic RL: the inability to optimize for autonomy during the training process when skills are present. By converting the "helpfulness" of a skill into a direct credit assignment signal for autonomous success, SKILLC eliminates the "internalization blindness" inherent in previous methods.

The authors emphasize that their approach does not require changing the underlying policy optimization framework (GRPO) but rather modifies the credit assignment mechanism. The significance lies in achieving autonomous performance that rivals skill-augmented methods without the runtime overhead of skill retrieval, all while providing a data-driven, adaptive mechanism for skill retirement.

Limitations Acknowledged:
The paper notes that SKILLC relies on fixed, pre-defined skill inventories and may struggle in open-ended settings with dynamic skill relevance. Additionally, the method incurs a computational overhead of approximately 26% in early training due to paired sampling, though this decreases as skills are retired. The effectiveness of the method is also contingent on the quality of the contrastive signal; in tasks where performance is already near ceiling, the signal may be too weak to drive further internalization.