Technical Summary: Decodable but Not Corrected by Fixed Residual-Stream Linear Steering

Problem Statement

This paper investigates a critical gap in the interpretability and control of Large Language Models (LLMs): the classification-correction gap. While prior work has demonstrated that specific behavioral states (e.g., refusal, sentiment) can be linearly decoded from hidden states and corrected via activation steering, it remains unclear whether this holds for complex, multi-step reasoning failures. The authors focus on a specific failure regime in medical question answering (MedQA) called Overthinking (OT), where models frequently produce correct answers under resampling but fail when generating extended chain-of-thought (CoT) traces. The central question is: If a failure mode is linearly decodable in the hidden states, can fixed linear steering vectors be used to correct it?

Methodology

1. Behavioral Regime Identification (Overthinking)

The authors define Overthinking (OT) as a behavioral construct where:

• The model answers correctly in $\ge$60% of sampled traces (indicating the knowledge is present).
• The model produces an incorrect answer in long-reasoning traces (>200 tokens).
• This regime is distinct from Knowledge Deficit (KD) (factual errors) and Reasoning Chain Break (RCB) (valid facts, invalid logic).
• OT is identified with high stability (Jaccard $\ge$0.81 under threshold perturbation, 94% inter-annotator agreement).

2. Geometric Analysis

Using Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct, the authors extracted hidden states from 12,000 traces (2,000 per failure mode + 6,000 matched correct traces).

• Contrastive Vectors: Computed as the normalized difference between mean hidden states of correct traces and specific failure modes ($v = \bar{h}_{correct} - \bar{h}_{mode}$).
• Classification: Linear probes were trained to distinguish OT from non-OT traces.
• Specificity Decomposition: The authors calculated the specificity ratio to determine how much of the contrastive vector is unique to the failure mode versus shared with general "incorrect-vs-correct" directions.
• Cross-Architecture & Cross-Domain: Validation was performed on Qwen2.5-7B and the MMLU-STEM domain to test generalizability.

3. Activation Steering Experiments

The authors tested five families of fixed linear steering interventions on the Llama-3.1-8B test set ($n=1,273$) and Qwen2.5-7B ($n=1,273$):

1. Contrastive Steering: Adding $\alpha \cdot v$ at peak layers.
2. Probe-Guided Steering: Using probe weight vectors instead of contrastive vectors.
3. Multi-layer Steering: Applying vectors at 1, 3, or 5 layers.
4. Rank-k Subspace Steering: Steering within subspaces spanned by top singular vectors.
5. Prompt-End Steering: Using vectors derived from the prompt (pre-generation) to avoid response contamination.

• Concept Erasure: A complementary intervention projecting out the failure direction to test causal entanglement.
• Controls: A refusal steering control verified the infrastructure; temperature sweeps and same-temperature evaluations ruled out distributional shifts as the primary cause of failure.

4. Post-Generation Reliability

The authors evaluated whether the decodable structure could support selective abstention (refusing to answer when the probe predicts low confidence) as an alternative to correction.

Key Results

1. The Classification-Correction Gap

• Decodability: OT is linearly decodable with 71.6% balanced accuracy ($p \approx 10^{-16}$) on Llama-3.1-8B. This signal is robust to length regression and threshold perturbations.
• Steering Failure: Despite high decodability, all 29 configurations of fixed linear steering across five method families yielded $\Delta \approx 0$ (no significant improvement in accuracy).
  • Mode-specific steering: $\Delta = -0.2$ pp.
  • Multi-layer steering: $\Delta \approx 0$.
  • Prompt-end steering: $\Delta \approx 0$.
• Cross-Architecture Consistency: The null result held for Qwen2.5-7B (9 configurations) and MMLU-STEM (4 configurations).

2. Evidence of Representational Entanglement

The failure to steer is attributed to representational entanglement between the failure signal and task-critical computation:

• Low Specificity: The OT-specific specificity ratio is only 0.119 (Llama) and 0.152 (Qwen). This means ~88% of the contrastive signal aligns with the shared "incorrect-vs-correct" direction.
• Causal Damage:
  • Shared-direction steering (applying the shared incorrect direction) severely damaged accuracy ($\Delta = -12.1$ pp).
  • Concept Erasure: Removing the OT direction via mean-difference projection damaged accuracy ($\Delta = -3.6$ pp, $p=0.01$), whereas 10 random directions produced no damage ($\Delta = +0.3$ pp). This confirms the OT direction is causally entangled with correct reasoning, not merely correlational.
• Per-Instance Uncorrelation: The correlation between probe confidence and steering effect is $r = -0.002$ ($p=0.97$), indicating that knowing when a model is overthinking does not predict how to steer it.

3. Positive Finding: Selective Abstention

While steering failed, the decodable structure enabled post-generation reliability estimation:

• A correctness probe achieved an AUROC of 0.610 on the held-out test set, significantly outperforming five single-forward-pass uncertainty baselines (best baseline AUROC = 0.569, $\Delta = 0.041$, $p=0.009$).
• Selective Abstention: By abstaining on low-confidence predictions, the model achieved +5.5 pp accuracy gain at 60% coverage.
• Comparison to Self-Consistency: While majority vote (Self-Consistency) corrects 100% of OT errors, it requires 10x generation cost. The probe offers a cost-effective single-pass alternative for reliability estimation.

4. Non-Linear Interventions

A preliminary experiment using a learned non-linear MLP (residual adapter) achieved a modest but significant improvement ($\Delta = +2.8$ pp, $p=0.025$). This suggests the gap is not absolute but specific to fixed linear interventions; non-linear transforms may partially disentangle the shared subspace.

Significance and Claims

The paper claims to provide mechanistic evidence for why activation steering fails in multi-step reasoning regimes, moving beyond mere documentation of the phenomenon.

1. Geometric Explanation: The authors argue that the classification-correction gap arises because failure signals in reasoning tasks are entangled with task-relevant computation. Unlike refusal (which has a specificity ratio of 0.999 and is steerable), the OT signal shares ~88% of its direction with the correct reasoning path.
2. Observation-Intervention Asymmetry: The study demonstrates that decodability does not imply steerability. A signal can be statistically readable by a probe (observation) without being causally actionable via additive perturbation (intervention) because the perturbation inevitably disrupts the shared task computation.
3. Operational Value of Probing: Even when steering fails, the decodable structure has utility for reliability estimation (selective abstention), offering a practical tool for deployment where correction is impossible.
4. Diagnostic Framework: The authors propose a Linear Accessibility Profile (LAP) and specificity ratio as pre-intervention diagnostics. They suggest that if a direction has low specificity (<0.21) and high overlap with task computation, fixed linear steering is unlikely to succeed.

The paper concludes that while the "classification-correction gap" is a robust empirical phenomenon in medical reasoning, it may be partially closable by learned non-linear interventions that can disentangle shared directions, a direction for future work.