Behavioral Steering in a 35B MoE Language Model via SAE-Decoded Probe Vectors: One Agency Axis, Not Five Traits

1. Problem Statement

The paper addresses the challenge of mechanistic interpretability and behavioral steering in large-scale, production-ready language models with non-standard architectures.

• Scale & Architecture Gap: Previous steering methods (Sparse Autoencoders/SAEs and Activation Engineering) have primarily been demonstrated on smaller models (up to 7B) with standard Transformer architectures. It remains unclear if these methods scale to massive Mixture-of-Experts (MoE) models with hybrid architectures (combining attention and linear recurrence layers like GatedDeltaNet).
• The "Probe vs. Causation" Gap: While linear probes can identify features correlated with specific behaviors, it is unknown whether these features are causally upstream of the behavior or merely epiphenomenal (correlated but not causal).
• Granularity: Existing methods often rely on clamping discrete SAE features (top-k), which destroys fine-grained directional information. The authors seek a method to steer behavior continuously without retraining.

2. Methodology

The authors propose a pipeline to train SAEs on a 35B model and use them to construct continuous steering vectors.

A. Model Target

• Model: Qwen 3.5-35B-A3B (35B total parameters, ~3B active per token).
• Architecture: Hybrid MoE with 40 layers. Each block consists of 3 GatedDeltaNet layers (linear recurrence) followed by 1 standard Attention layer.
• Residual Stream: Hidden dimension $d=2048$.

B. SAE Training

• Training: Nine top-k SAEs were trained on residual stream activations at various depths (early, mid, late) covering both GatedDeltaNet and Attention sublayers.
• Data: 200M tokens from a mix of chat datasets and synthetic tool-use conversations.
• Technique: Used the FAST methodology with auxiliary dead-feature losses to ensure training stability.

C. SAE-Decoded Probe Steering (The Core Innovation)

Instead of directly clamping SAE features (which are discretized by the top-k function), the authors use a two-step projection:

1. Linear Probe: Train a ridge regression probe on SAE latent activations ($z$) to predict a behavioral trait (High vs. Low contrastive pairs). This yields probe weights $w_{probe}$.
2. Decoder Projection: Project these weights back through the SAE decoder ($W_{dec}$) to obtain a continuous steering vector in the model's native residual stream space:
   $$v_{steer} = W_{dec}^{\top} w_{probe}$$
   • Benefit: This bypasses the SAE's top-k discretization, allowing for fine-grained, continuous intervention at any scale $\alpha$.

D. Evaluation

• Setup: 1,800 agent rollouts across 50 ReAct-style scenarios (coding, research, etc.).
• Metrics: Behavioral proxies extracted from trajectories (e.g., frequency of ask_user vs. proactive tool calls).
• Conditions: Tested steering at different multipliers ($\alpha$) and application modes: all positions, prefill only, and decode only.

3. Key Contributions

1. Scalability: Demonstrated that SAE-decoded probe steering works on a 35B hybrid MoE model, achieving significant effect sizes without retraining.
2. The "One Agency Axis" Discovery: Cross-trait analysis revealed that steering vectors for five distinct traits (Autonomy, Tool-use, Persistence, Risk Calibration, Deference) do not operate independently. Instead, they all primarily modulate a single dominant agency axis: the disposition to act independently vs. defer to the user. Trait-specificity manifests only in secondary properties (e.g., which tool is used, not whether to act).
3. Correlation vs. Causation Dissociation: Provided empirical evidence that high probe accuracy ($R^2$) does not guarantee causal efficacy. Two traits (Risk Calibration and Tool-Use) probed from the same SAE with nearly identical $R^2$ yielded orthogonal vectors; one successfully steered behavior, while the other produced only suppression.
4. Prefill-Only Behavioral Commitment: Showed that steering applied only during autoregressive decoding has zero effect. Behavioral decisions are computed and committed during the prefill phase and propagated through the GatedDeltaNet recurrent state.

4. Key Results

A. Steering Efficacy

• Autonomy: Achieved a large effect size ($d = 1.01$) at $\alpha=2$. The model shifted from asking for help (78% of the time) to proactively executing code and searching the web.
• Dose-Response Taxonomy:
  • Smooth Inverted-U: Autonomy and Deference (clear therapeutic window).
  • Phase Transition: Tool-use eagerness (intermediate $\alpha$ degrades performance; high $\alpha$ triggers a new behavioral mode).
  • Pure Suppression: Persistence and Risk Calibration (increasing $\alpha$ suppresses behavior without replacing it).

B. The Risk-Calibration Dissociation

• Observation: Risk Calibration and Tool-Use vectors share the same SAE (Layer 21) and have nearly identical probe $R^2$ (0.795 vs. 0.792).
• Geometry: The vectors are nearly orthogonal (cosine similarity $\approx -0.017$).
• Outcome: The Tool-Use vector successfully steered behavior ($d=0.39$), while the Risk-Calibration vector produced zero causal effect ($d=0.00$), only suppressing existing behavior.
• Implication: The Risk-Calibration direction is a statistical epiphenomenon—it predicts the trait but is not causally upstream of it.

C. Prefill vs. Decode

• Applying steering vectors only during decoding resulted in no behavioral change ($p > 0.35$).
• Applying steering only during prefill yielded significant results (especially for Deference).
• Conclusion: In GatedDeltaNet architectures, the "decision" to act or defer is compiled into the recurrent state during the prompt processing (prefill) phase.

D. Feature Concentration

• Steering vectors are extremely sparse. For example, 48 features (0.3% of the dictionary) explain 50% of the Autonomy steering signal. This concentration may explain why these interventions are robust against the model "routing around" them.

5. Significance and Implications

• Architectural Insight: The findings suggest that hybrid architectures (GatedDeltaNet + Attention) compute high-level behavioral commitments early (prefill) and propagate them via recurrence, rather than refining them token-by-token during generation.
• Interpretability Caution: The paper challenges the reliance on linear probes for steering. High $R^2$ is necessary but insufficient for causal intervention. Future work must validate probes with intervention experiments.
• Safety & Control: The discovery of a "dominant agency axis" implies that attempting to steer a specific trait (e.g., "be more helpful") might inadvertently shift the model's entire disposition toward independence, potentially bypassing safety-aligned deference behaviors.
• Deployment Strategy: Effective behavioral steering requires prefill-phase intervention and trait-specific calibration (therapeutic windows vary by trait), rather than a universal scaling parameter.

6. Limitations

• Sample Size: 50 scenarios per condition provides power for large effects but may miss medium effects.
• Proxy Metrics: Reliance on tool-call counts (ask_user vs. others) may conflate correlation with causation in the cross-trait matrix.
• Generalizability: Results are specific to Qwen 3.5-35B-A3B; generalization to pure Transformers or other MoE models is unverified.
• Baseline: No direct comparison against simple mean-difference steering (without SAEs) was performed.

In summary, the paper successfully scales behavioral steering to a 35B hybrid model, revealing that agentic behavior is organized around a single "agency" bottleneck rather than five independent traits, and highlighting a critical gap between predictive correlation and causal control in neural representations.