Technical Summary: Universal Boosts, Specific Suppressors: Sparse Autoencoder Steering of Medical Vision-Language Models

Problem Statement

Medical Vision-Language Models (VLMs) frequently hallucinate findings when generating radiology reports from chest X-rays. These errors manifest as fabricating non-existent findings, omitting critical pathologies, or mislocalizing/misassessing the severity of observed conditions. While existing mitigation strategies rely on post-training optimization—such as Reinforcement Learning (RL) with Direct Preference Optimization (DPO) or Group Relative Policy Optimization (GRPO)—these approaches incur substantial computational overhead (e.g., hundreds of GPU-hours per model) and require gradient updates, optimized reward signals, and model-specific training runs. Furthermore, these "black box" fine-tuning methods often lack interpretability regarding which internal mechanisms drive specific clinical errors.

Methodology

The authors propose an inference-only approach that steers frozen medical VLMs by editing their internal activations at decoding time, utilizing Sparse Autoencoders (SAEs). The pipeline consists of four stages:

1. Residual-Stream Activation Collection: Per-token hidden states ($h^{(t)}_\ell$) are collected from specific transformer layers during report generation on a training/validation split.
2. Sparse Autoencoder Training: A Top-K SAE is trained at each selected layer to decompose the dense hidden states into high-dimensional sparse representations ($z^{(t)}_\ell$). The authors use a dictionary size of $D=32,768$ and sparsity $K=64$.
3. Feature Identification (Causal Screening):
   • Prefiltering: Features are ranked by the absolute gap in mean activation between high-error (top quartile) and low-error (bottom quartile) samples.
   • Causal Screening: A single-feature intervention is performed on a held-out validation set. For each candidate feature, the authors ablate it (zero it out) or amplify it and measure the resulting change in automatic clinical metrics, specifically the GREEN error taxonomy (False Findings, Missing Findings, Wrong Location, Wrong Severity).
   • Classification: Features are categorized as suppress candidates (ablation reduces errors) or boost candidates (ablation increases errors, meaning the feature promotes quality). Crucially, features are identified per-error-type, acknowledging that a single feature may suppress one error type while boosting another.
4. Residual Steering: At inference time, the identified features in the SAE basis are edited: suppress features are zeroed, and boost features are scaled by $(1+\beta)$. Instead of replacing the hidden state with the SAE reconstruction (which introduces reconstruction error), the authors apply a pure residual update:
   $$\delta_\ell = \text{decode}_\ell(z'_\ell) - \text{decode}_\ell(z_\ell)$$
   $$h'_\ell = h_\ell + \alpha \delta_\ell$$
   This ensures that only the specific delta caused by the intervention is added to the original hidden state, preserving the model's unedited knowledge.

Key Contributions

• Inference-Only Mitigation: The paper demonstrates that hallucinations can be mitigated without weight updates, retraining, or prompt engineering, significantly reducing computational costs compared to RL fine-tuning.
• Causal Feature Discovery: The authors identify that clinically meaningful SAE features are not monolithic "hallucination directions." Instead, they exhibit a complex structure where:
  • Boost features (promoting report quality) are largely universal across different architectures (RadVLM, LLaVA-Rad, CheXOne), showing high functional overlap (cosine similarity ~0.91–0.95).
  • Suppress features (reducing specific errors) are model-specific, with low functional overlap (cosine similarity as low as 0.25). This implies that while quality-promoting mechanisms are shared, the internal pathways leading to hallucinations are unique to each model's training regime.
• Granular Error Control: The method allows for separate steering lists for different error types (e.g., Missing Findings vs. False Findings), enabling targeted interventions that address the specific causal roles of features.
• Zero-Shot Transfer: The steering recipe, identified on MIMIC-CXR, transfers effectively to the independent IU-Xray dataset without retraining, confirming that the identified features are properties of the model architecture rather than artifacts of the training corpus.

Results

The method was evaluated on three medical VLMs (RadVLM, LLaVA-Rad, and CheXOne) on the MIMIC-CXR test split, with additional zero-shot evaluation on IU-Xray.

• Performance Gains:
  • RadVLM: +5.4% relative improvement in the clinical composite metric; +3.9% absolute improvement in GREEN score.
  • LLaVA-Rad: +7.2% relative improvement in composite metric; +10.2% absolute improvement in GREEN.
  • CheXOne: +17.0% relative improvement in composite metric; +11.7% absolute improvement in GREEN. Notably, gains were observed even on CheXOne, which was already fine-tuned with GRPO, suggesting SAE steering is complementary to RL.
• Error Redistribution: The steering mechanism primarily recovers Missing Findings (reducing them by 654 in RadVLM) at the cost of a moderate increase in False Findings (+1,112). Due to the structural asymmetry of the GREEN metric (where recovering a missing finding has a larger positive impact than adding a false finding has a negative impact), the net result is a significant improvement in overall report quality and clinical utility.
• Cross-Model Alignment: Analysis confirmed that boost features are highly consistent across models, whereas suppress features are distinct. This supports the conclusion that transferable steering must treat suppression on a per-backbone basis.

Significance and Claims

The paper claims that its primary contribution is providing a mechanistic window into medical VLMs. By decomposing the residual stream into interpretable, sparse coordinates, the authors can:

1. Name and locate the specific internal mechanisms responsible for clinical errors.
2. Edit these mechanisms at inference time to improve safety and accuracy without retraining.
3. Understand that hallucination mitigation is not a single global direction but a structured trade-off between completeness (boosting) and specificity (suppressing).

The authors emphasize that while the absolute metric gains are moderate compared to heavy RL fine-tuning, the approach offers a distinct advantage in interpretability and efficiency. It avoids the "black box" nature of RL by exposing the causal circuits of the model, allowing for targeted, transparent interventions. The work does not claim to replace radiologists or serve as a standalone clinical decision-making tool but rather as a research framework for improving the reliability of automated report generation.