Technical Summary: Knowledge is Not Enough: Injecting RL Skills for Continual Adaptation

Problem Statement

Large Language Models (LLMs) face a fundamental "knowledge cutoff" challenge: their parametric memory is frozen after pre-training, preventing the native internalization of new information or tools. While Retrieval-Augmented Generation (RAG) attempts to mitigate this via external context, it suffers from long-range dependency modeling issues and inference overhead. Consequently, research has shifted toward Parametric Knowledge Updating (e.g., Knowledge Editing, Test-Time Training).

However, a critical limitation exists in current adaptation paradigms: a functional disconnect between knowledge and reasoning skills.

• Supervised Fine-Tuning (SFT) effectively memorizes factual content (the "what") but often fails to teach the model how to reason over or manipulate that knowledge in downstream tasks. Models trained via SFT may "know" facts but fail to utilize them dynamically, leading to hallucinations or rigid behavior when encountering errors.
• Reinforcement Learning (RL) is essential for acquiring robust reasoning and execution skills (the "how"), but its high computational cost (requiring on-policy exploration and interaction data) makes it impractical for efficient, continual online adaptation to every new scenario.

The core problem is how to equip a model with new domain knowledge and the procedural skills to use it, without incurring the prohibitive cost of training RL from scratch for every new domain.

Methodology: Parametric Skill Transfer (PaST)

The authors propose Parametric Skill Transfer (PaST), a modular framework that decouples knowledge acquisition from skill acquisition, allowing for the transfer of reasoning capabilities across domains without explicit RL in the target domain.

1. Theoretical Basis: Orthogonality of Updates

The method relies on the empirical observation that parameter updates induced by SFT ($\Delta W_{SFT}$) and RL ($\Delta W_{RL}$) are nearly orthogonal.

• Empirical Evidence: Cosine similarity analysis across model layers (e.g., on LooGLE and ToolBench tasks) shows that $\Delta W_{SFT}$ and $\Delta W_{RL}$ reside in disentangled subspaces of the parameter landscape. Their correlation is consistently near zero, whereas two independent SFT updates show significant correlation.
• Implication: Knowledge (factual injection) and Skills (procedural logic) can be optimized separately and linearly composed without destructive interference.

2. The PaST Framework

PaST operates in two stages:

Stage I: Source Domain Skill Extraction

1. SFT Anchoring: A base model is fine-tuned on a source domain's knowledge corpus ($C_S$) to obtain $\theta^S_{sft}$.
2. RL Refinement: The SFT model undergoes RL training on source interaction trajectories ($T_S$) to internalize reasoning policies, resulting in $\theta^S_{rl}$.
3. Skill Vector Extraction: A domain-agnostic Skill Vector ($v_{skill}$) is extracted by subtracting the SFT parameters from the RL parameters:
   $$v_{skill} = \theta^S_{rl} - \theta^S_{sft}$$
   This vector captures the "gradient" direction of reasoning improvement, stripped of domain-specific declarative facts.
   • Refinement: To prevent overfitting to specific source content, an Iterative Bootstrapping Strategy is employed. The source domain is split into subsets; the skill vector is refined iteratively across rounds, injecting the previous round's vector as a warm-start for the next RL phase.

Stage II: Target Domain Adaptation

1. Lightweight SFT: The target model is fine-tuned on new target domain documents ($C_T$) to obtain $\theta^T_{sft}$. This anchors the model with the new facts.
2. Linear Injection: The extracted skill vector is linearly injected into the target model:
   $$\theta_{final} = \theta^T_{sft} + \lambda \cdot v_{skill}$$
   (where $\lambda$ is a scaling coefficient, set to 1 in experiments). This grafts the source-learned reasoning geometry onto the target knowledge manifold, enabling zero-shot execution of complex tasks.

Key Contributions

1. Identification of Reasoning-Knowledge Disconnect: The paper highlights that SFT alone is insufficient for transferring procedural logic to new domains, as it primarily induces surface-level memorization.
2. Empirical Evidence of Orthogonality: The authors provide rigorous evidence that SFT and RL updates occupy nearly orthogonal subspaces, validating the feasibility of separating and recombining knowledge and skills.
3. PaST Framework: A novel method utilizing task vector arithmetic to transfer RL-learned skills from a source domain to a target domain, bypassing the need for expensive target-domain RL.
4. Decoupling and Transferability: Demonstration that knowledge manipulation and execution skills can be effectively decoupled and transferred, enabling robust adaptation in data-scarce target domains.

Experimental Results

The authors evaluated PaST across three benchmarks:

1. Knowledge Incorporation (SQuAD):

   • Setting: Closed-book QA where the model must memorize a passage via test-time updates and answer questions without context.
   • Result: PaST achieved 56.9% accuracy, significantly outperforming the state-of-the-art self-editing baseline SEAL (47.0%) and a standard "Train on Passage + Synthetic" baseline (39.7%). This represents a +17.2% absolute improvement over the synthetic baseline, proving the skill vector provides essential inference logic.

2. Long-Context Reasoning (LooGLE):

   • Setting: QA on documents exceeding 21k tokens.
   • Result: PaST improved accuracy from 30.1% (Standard SFT) to 38.1% (+8.0 points). The method successfully scaled to massive documentation, transforming the model from a passive fact container into an expert capable of precise parametric retrieval.

3. Cross-Domain Tool Use (ToolBench):

   • Setting: Closed-book execution where the model must use APIs based only on names (no context), transferring skills from a "Movies" source domain to 20 unseen target categories.
   • Result: PaST increased the average success rate from 21.9% (Target SFT) to 32.2% (+10.3 points). It achieved zero-shot activation in domains where the baseline failed completely (e.g., Advertising: 0% $\to$ 16.7%).

4. Efficiency Comparison:

   • PaST matched the performance of target-domain RL trained for ~75 steps (44.0% vs 44.6%) but incurred zero additional training time on the target domain. Target RL required ~72 minutes per document, whereas PaST required none.

Significance and Claims

The paper claims that PaST offers a computationally efficient and scalable alternative to on-policy RL for continual adaptation. By leveraging the orthogonality of parameter updates, the method allows models to acquire new knowledge via lightweight SFT while simultaneously inheriting robust reasoning capabilities from a pre-trained source. This bridges the gap between "knowing" facts and "using" them, addressing a critical bottleneck in deploying LLMs in dynamic, real-world environments where both knowledge and reasoning skills must evolve rapidly.

The authors modestly note limitations, including the need for broader experimental domains beyond QA and tool use, the use of a static scaling coefficient ($\lambda=1$), and the primary reliance on the Qwen2.5-7B architecture, suggesting future work is needed to confirm these dynamics across broader model scales.