Extraction and Analysis of Multimodal Concepts in Vision Language Models through Sparse Autoencoders

Technical Summary: Extraction and Analysis of Multimodal Concepts in VLMs through Sparse Autoencoders

Problem Statement
Vision Language Models (VLMs) have achieved significant performance in joint image-text tasks like Visual Question Answering (VQA) and image captioning. However, understanding their internal computational mechanisms remains a critical challenge. While Sparse Autoencoders (SAEs) have emerged as a promising tool for Mechanistic Interpretability (MI) to extract human-understandable concepts from model activations, existing approaches suffer from two primary limitations when applied to VLMs:

1. Modality Isolation: Most SAE-based works focus exclusively on textual concepts or visual concepts in isolation, ignoring multimodal concepts (concepts jointly grounded in both image and text). This leads to potential misclassification of neurons that encode integrated information.
2. Low-Quality Visual Descriptions: Previous visual concept extraction methods often produce vague or incomplete descriptions. Furthermore, techniques relying on masking image patches to isolate neuron activations remove too much contextual information, degrading the accuracy of the extracted concepts.

Methodology
The authors propose a unified framework based on SAEs to extract and analyze visual, textual, and multimodal concepts. The methodology proceeds as follows:

1. SAE Integration and Activation Recording:

   • A pretrained SAE is integrated into a VLM (specifically LLaVA-NeXT-8B).
   • The SAE projects high-dimensional embeddings into a sparse latent space ($\omega$) using a Top-K activation function to enforce sparsity and encourage monosemantic neurons.
   • For each VQA sample (image + question-answer pair), the system records separate activation tensors for visual patches ($V$) and textual tokens ($T$).

2. Multimodal Concept Hypothesis Extraction:

   • For a target SAE neuron, the system selects the top-5 samples that maximize the average activation for that neuron in both visual and textual modalities.
   • Visual Extraction: An external VLM (LLaVA-72B) acts as an "explainer." It receives the top-activating image patches (with non-activating patches blurred, rather than black-masked, to preserve context) alongside the corresponding VQA question and answer. This text-augmented visual input improves the explainer's understanding.
   • Textual Extraction: The explainer receives the specific textual tokens that triggered the neuron (highlighted in brackets) alongside the original unmodified image. This mirrors the visual process, providing multimodal grounding for the text.
   • The explainer generates a candidate human-interpretable concept hypothesis for each modality.

3. Evaluation and Classification:

   • Hypothesis-Data Alignment: To validate hypotheses, the framework uses CLIP (ViT-B/32) and ALIGN models to compute cosine similarity between the generated concept descriptions and the data samples that triggered them. A hypothesis is considered validated if it ranks highly (top-$n$) against its generating data.
   • Multimodal Neuron Assignment: For neurons with both visual ($h_{vis}$) and textual ($h_{text}$) hypotheses, the framework computes the semantic similarity between the two hypotheses using text encoders. If the similarity exceeds a threshold $\tau$ (set to 0.8), the neuron is classified as multimodal. Otherwise, it is assigned to the modality with the higher hypothesis-data alignment score.
   • Category-Based Evaluation: Visual hypotheses are also mapped to specific visual categories (e.g., scene, object, part) to compare against baselines.

Key Contributions
The paper presents two primary contributions:

1. Integrated Multimodal Concept Extraction: The framework extends SAE analysis to a unified setting, successfully extracting and distinguishing between visual, textual, and multimodal concepts. It addresses the gap in systematic studies of multimodal concepts in VLMs.
2. Improved Visual Concept Quality: By introducing a new extraction technique that includes complementary text (Q&A pairs) during visual hypothesis generation and replacing black masking with blurring, the method yields more precise and semantically grounded visual concepts.

Experimental Results
Experiments were conducted on the LLaVA-NeXT dataset (using the first 5,000 neurons of the SAE) and compared against a baseline method (Zhang et al. [21]).

• Visual Concept Quality: The proposed framework significantly outperformed the baseline. It achieved an average improvement of 41% in CLIP scores and 45% in ALIGN scores for visual concept alignment across various thresholds.
• Multimodal Identification: The framework successfully identified 132 multimodal concepts out of 249 analyzed neurons (at $\tau=0.8$), whereas the baseline did not systematically identify multimodal concepts.
• Textual Quality: The quality of textual concepts remained high, with interpretability scores (detection and fuzzing) comparable to other textual-only SAE analysis works (e.g., using Llama-3.1-8b).
• Qualitative Intervention: Intervention experiments demonstrated that forcing specific neuron activations (visual, textual, or multimodal) influenced text generation. Notably, multimodal concepts appeared to have a more pronounced impact on output production than unimodal concepts, especially when an image was present in the prompt.

Significance and Claims
The authors claim that this work provides new insights into the conceptual space of VLMs by demonstrating that SAEs encode textual and visual concepts condensed into a shared latent space. The framework offers a structured approach to distinguish between unimodal and multimodal concepts, which is essential for understanding how VLMs align and process inputs across modalities.

The paper concludes that their approach enables a more comprehensive understanding of VLM reasoning by improving the quality of extracted visual concepts and systematically identifying multimodal neurons. The authors note that while their findings reveal a correlation between concept alteration and text output, they do not establish a causal relationship. They position this work as a step toward scalable SAE integration and concept-guided control in multimodal reasoning systems, without claiming immediate deployment in production systems or broader applications beyond the scope of interpretability research.