🤖 machine learning

Localizing RL-Induced Tool Use to a Single Crosscoder Feature

This paper demonstrates that Dedicated Feature Crosscoders (DFC) can isolate a compact set of RL-induced features in Qwen2.5-3B that not only significantly improve tool-calling correctness but also enable the transfer of these agentic capabilities to a frozen base model, facilitating retraining-free behavioral control.

Original authors: Andrii Shportko, Shubham Bhokare, Ahmed Zeyad A Alzahrani, Bowen Cheng, Gustavo Mercier, Jessica Hullman

Published 2026-06-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Andrii Shportko, Shubham Bhokare, Ahmed Zeyad A Alzahrani, Bowen Cheng, Gustavo Mercier, Jessica Hullman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart robot (a Large Language Model) that knows how to talk, write, and answer questions. But it doesn't know how to use tools, like calling a calculator or searching the web. You teach it this new skill using a method called Reinforcement Learning (RL), which is like giving the robot a reward every time it uses a tool correctly.

After this training, the robot is great at using tools. But here's the mystery: Where exactly in the robot's brain did this new skill live? Did it change the whole brain, or is there a tiny, specific switch that got flipped?

This paper tries to find that switch. Here is how they did it, explained simply:

1. The Problem: A Messy Brain

When you train a robot, its internal "brain" (mathematical representations) gets messy. It's hard to tell which part of the brain is responsible for the new tool skills and which part is just the robot's original personality.

2. The Tool: A Special "X-Ray" Machine

The researchers built a special tool called a Dedicated Feature Crosscoder (DFC). Think of this as a high-tech prism or a prism-like filter.

They took the original robot (Model B) and the trained robot (Model A).
They shined their "thoughts" through this prism.
The prism splits the thoughts into three piles:
1. The "Original" Pile: Things only the original robot does.
2. The "Shared" Pile: Things both robots do.
3. The "Exclusive" Pile: Things only the trained robot does (the new tool skills).

3. The Big Discovery: The "Magic Switch"

The researchers expected the new tool skills to be spread out or mixed in with the shared pile. Instead, they found something amazing:

The new skill was concentrated into just ONE tiny switch (a single feature).

The Analogy: Imagine the robot's brain is a giant library with millions of books. You might think learning to use a tool requires reading 10,000 new books. But this paper found that the entire skill of "using tools" was stored in one single book.
The Proof: When they turned only that one specific switch "on" in the original, untrained robot, the robot suddenly started using tools correctly! They didn't need to retrain it. They just flipped that one switch, and the robot gained the ability.

4. The "Spillover" Effect

Here is a funny side effect they noticed. Because they trained the two robots together using this prism, the "idea" of using tools leaked out.

Even though they only turned the switch on for the trained robot, the original robot (which never saw the training) also got slightly better at using tools just by being part of the process.
Analogy: It's like two people studying for a test together. One person learns the material perfectly. The other person, just by sitting in the same room and looking at the same notes, accidentally picks up a little bit of the knowledge, even though they didn't study hard.

5. Why This Matters

The paper shows that we don't need to retrain a robot to change its behavior. We can just find the specific "switch" that controls a behavior (like using a tool) and flip it.

Before: To fix a robot, you might have to retrain it from scratch (like going back to school).
Now: You can just find the one switch and flip it (like turning on a light).

Summary of Results

They tested 48 different versions of their "prism" tool to make sure it worked.
They found one specific switch that, when flipped, made the robot's tool-use accuracy jump by 65%.
They proved that this switch is unique to the trained robot and doesn't exist in the original one.
They showed that if you try to mix this switch with other "shared" parts, it actually makes things worse (like trying to fix a watch with a hammer).

In short: The researchers found that complex new behaviors in AI aren't scattered everywhere. They are often hidden in a single, tiny, controllable switch. If you find that switch, you can control the robot's behavior instantly without any extra training.

Technical Summary: Localizing RL-Induced Tool Use to a Single Crosscoder Feature

Problem Statement

While Reinforcement Learning (RL) fine-tuning effectively enables Large Language Models (LLMs) to perform agentic behaviors, such as structured tool invocation, the mechanistic basis of these internal representation changes remains poorly understood. Specifically, it is unclear which features emerge during RL, which are preserved, and whether these features can be isolated to enable retraining-free behavioral control. Existing mechanistic interpretability tools, such as Sparse Autoencoders (SAEs) and Crosscoders, offer frameworks for decomposing model activations, but their ability to isolate capability-specific differences between a base model and an RL-fine-tuned model has not been systematically evaluated.

Methodology

The authors investigate these questions using a Dedicated Feature Crosscoder (DFC) architecture applied to a pair of models sharing the Qwen2.5-3B architecture:

Model A (RL): A ToolRL-fine-tuned variant capable of structured tool invocation.
Model B (Base): The original, unmodified Qwen2.5-3B.

Architecture and Training:
The DFC partitions a joint sparse dictionary into three sub-dictionaries: A-exclusive, B-exclusive, and shared. Gradient masking enforces exclusivity, forcing each model to read from and write to only its designated partition. The training objective minimizes reconstruction loss (MSE) while applying sparsity constraints (Top-k) and specific penalties ( $\lambda_{excl}$ ) to the exclusive partitions to test whether they act as "sinks" (isolating all capability differences) or "filters."

Experimental Design:

Hyperparameter Sweep: The authors trained and evaluated 48 variants (36 DFCs and 12 standard Crosscoders) across a systematic sweep of dictionary sizes ( $D$ ), sparsity budgets ( $k$ ), exclusive partition shares ( $p$ ), and penalty magnitudes ( $\lambda_{excl}$ ).
Evaluation Metrics: Performance was measured via tool_correctness (fuzzy matching called tools to the prompt) and format_accuracy (presence of correct JSON structure).
Steering Protocol: Features were ranked by Cohen's $d$ (difference in activation between tool-use and general-text contexts). Targeted steering involved adding a correction vector to the residual stream of Model A based on specific feature subsets ( $S$ ).

Key Contributions and Results

1. Capability Spillover and Reconstruction Fidelity

The study demonstrates that capability spillover occurs: passing the frozen base model's (Model B) activations through a jointly trained crosscoder and reconstructing them passively transfers tool-calling ability without fine-tuning.

Result: Model B's tool correctness increased from 0% to 6.8% ± 5.0 pp post-reconstruction.
Implication: Tool-calling intent is delocalized into shared decoder weights during joint training. However, format accuracy (the specific XML/JSON surface form) did not spill over, suggesting that while semantic intent is shared, the structural template machinery remains isolated in narrow RL-specific features.

2. DFC as a Filter, Not a Sink

The authors tested whether the exclusive partition perfectly isolates RL-induced capabilities (a "sink").

Finding: Penalizing the exclusive partition ( $\lambda_{excl} > 0$ ) degraded Model A's fidelity, pushing tool-relevant information back into the shared partition.
Conclusion: The exclusive partition acts as a filter that concentrates the strongest model-specific features but does not fully disentangle capability-specific content from the shared representational substrate.

3. Single-Feature Steering Saturation

A primary finding is that RL-induced tool-use capability can be localized to a minimal set of features, often a single neuron.

Result: Steering a single A-exclusive neuron ( $|S|=1$ ) achieved a +65.0 pp increase in tool correctness (95% CI [+47.9, +82.1]).
Comparison: This single-neuron intervention matched the performance of steering all 7 available A-exclusive neurons. In contrast, a standard CrossCoder required steering 33 features to reach its peak performance.
Generalization: This single-neuron saturation effect generalized across the majority of probed layers (7 out of 9), with a mean improvement of +43.3 pp.

4. Geometric Separation

UMAP analysis of decoder weights confirmed that the DFC architecture creates spatially distinct regions for A-exclusive, B-exclusive, and shared features. In contrast, standard Crosscoders (without gradient masking) showed uniform mixing of biased features, indicating that the clean separation is an artifact of the architectural partitioning, not merely label imbalance.

5. Interference in Combined Steering

Steering both A-exclusive and shared features simultaneously resulted in destructive interference, yielding lower performance (+35.0 pp) than steering A-exclusive features alone (+65.0 pp). This suggests that non-orthogonal decoder directions amplify noise when steered with equal coefficients.

Significance and Claims

The paper claims that Dedicated Feature Crosscoders provide a promising tool for identifying and modulating representations introduced by RL post-training. The significance lies in three main areas:

Runtime Behavioral Control: The ability to steer agentic LLMs via a single feature without retraining offers a gradient-free, inference-time control mechanism. This positions DFC-based steering as a complementary approach to RLHF, DPO, and SFT for safety and behavior modification.
Mechanistic Interpretability: The findings suggest that agentic capabilities (specifically tool use) can be compressed into a sparse, steerable feature set. The "capability spillover" phenomenon highlights that joint sparse decompositions can inadvertently reintroduce capabilities into base models, a critical consideration for model safety and release.
Limitations and Modesty: The authors explicitly note that their metrics measure the propensity to emit correct tool calls under fixed prompts, not the installation of a wholly new ability in the base model. They acknowledge that the architecture-effect t-test is underpowered (12 CrossCoder runs vs. 36 DFC runs) and that the best-cell steering results have not been replicated on a fresh sample. They do not claim that RL installs a tool-use ability the base model wholly lacks, but rather modulates a propensity.

In summary, the work demonstrates that RL-induced tool use is not a diffuse, uninterpretable change but is concentrated in a minimal, steerable feature set that can be isolated and manipulated via DFCs, enabling precise, retraining-free behavioral control.