Technical Summary: Vision-Language Models Align with Human Neural Representations in Concept Processing

Problem Statement

Recent studies suggest that transformer-based Vision-Language Models (VLMs) capture the multimodality of concept processing in the human brain. However, a systematic evaluation exploring different VLM architectures and the specific roles of visual versus textual context is lacking. Existing literature presents diverging conclusions: some studies indicate multimodal advantages, while others suggest multimodal fine-tuning may harm human-alignment properties acquired by language-only models. Furthermore, previous work often fails to distinguish between the benefits of accessing visual information at inference versus genuinely learning human-like multimodal representations during pretraining. Additionally, most studies focus on the multimodal/unimodal dichotomy rather than the substantial differences between specific VLM families (contrastive, encoders, and generative).

Methodology

The authors conducted a systematic comparison of ten models, including six VLMs and four language-only counterparts, against human fMRI data from the Pereira et al. (2018) dataset. The study utilized Representational Similarity Analysis (RSA) to measure the alignment between model representations and brain responses to 180 concept words.

Experimental Conditions:
The study evaluated models under two distinct conditions to isolate the effects of context:

1. Sentence Condition: Concept words were presented within sentences. Models received text-only input (with LXMERT receiving random noise images to satisfy architectural requirements).
2. Picture Condition: Concept words were presented alongside images illustrating the concept. VLMs received both text and image inputs; language-only models received text only.

Model Families:
The study covered three VLM families and their language-only counterparts:

• Contrastive Models: CLIP, ALIGN.
• Vision-Language Encoders: VisualBERT, LXMERT.
• Generative VLMs: IDEFICS2, LLaVA-NeXT.
• Language-Only Counterparts: BERT, Mistral, Llama3, and GloVe.

Evaluation Metrics:

• RSA: Representational Dissimilarity Matrices (RDMs) were constructed using cosine distances between concept representations (for models) and voxel-wise fMRI activations (for brains). Alignment was quantified via Spearman correlation between model RDMs and brain RDMs.
• Brain Networks: Analysis focused on the left-hemisphere Language network and the Visual network.
• Ablation Studies:
  • Partial Correlation: To determine if VLM alignment stems from their underlying language modules or multimodal pretraining, the authors computed partial correlations removing information shared with the language-only counterparts.
  • Input Control: In the picture condition, VLMs were tested with text-only inputs (no images) to distinguish between alignment driven by inference-time context versus pretraining.

Key Results

1. General VLM Advantage:
VLMs generally outperformed language-only counterparts in both experimental conditions across both brain networks. This suggests VLMs successfully model human-like concept processing.

2. Architecture-Specific Performance:

• Vision-Language Encoders: Models like VisualBERT and LXMERT consistently exhibited the highest brain alignment, often outperforming more powerful generative VLMs.
• Generative VLMs: Despite their superior performance on downstream tasks, generative models (IDEFICS2, LLaVA) showed lower brain alignment compared to encoders. In the sentence condition, they were not significantly better than their language-only counterparts (Mistral, Llama3).
• Contrastive Models: CLIP and ALIGN showed moderate alignment, performing well in the picture condition but struggling in the sentence condition compared to encoders.

3. Context and Pretraining Insights (Ablation Studies):

• Source of Alignment: Partial correlation analysis revealed that for LXMERT and IDEFICS2, brain alignment stems from genuinely learning human-like concepts during multimodal pretraining, as their alignment persisted even when language-module information was controlled for. Conversely, for other models (e.g., VisualBERT, LLaVA), alignment was heavily dependent on the information shared with their language-only bases.
• Input Sensitivity: In the picture condition ablation (removing images), models like CLIP, ALIGN, VisualBERT, and LLaVA suffered dramatic drops in brain alignment, indicating their performance relied heavily on the visual context provided at inference. In contrast, LXMERT and IDEFICS2 maintained high alignment even without image inputs, suggesting robust internal multimodal representations.

4. Network Specifics:

• Language Network: VLMs generally outperformed language-only models. VisualBERT and LXMERT were significantly more aligned than BERT.
• Visual Network: Surprisingly, language-only models achieved significant correlations in the visual network even without visual input. However, VLMs (specifically VisualBERT and IDEFICS2) still achieved the highest correlations.

Significance and Claims

The paper claims to provide a broad investigation into how different VLM families align with human neural representations in concept processing. The primary significance lies in demonstrating that:

1. Multimodality is Beneficial: VLMs generally align better with human brain responses than unimodal models, supporting the hypothesis that semantic representations grounded in multimodal experiences approximate human cognition better.
2. Architecture Matters: Not all VLMs are equal. Vision-language encoders (e.g., LXMERT) appear to model brain responses better than recent generative VLMs, suggesting that performance on downstream generative tasks does not necessarily correlate with human-like concept processing.
3. Pretraining vs. Inference: The study distinguishes between models that learn multimodal representations during pretraining (e.g., LXMERT, IDEFICS2) and those whose alignment is primarily driven by the visual context provided at inference.
4. Contextual Nuance: The advantage of VLMs is modulated by the type of context (visual vs. sentential) and the specific architecture, highlighting that the "multimodal advantage" is not a monolithic property but depends on how modalities are integrated.

The authors conclude that their findings contribute to the debate on whether multimodality results in more human-like language models, showing it is beneficial for brain alignment in concept processing, while emphasizing the need to consider architectural differences. They explicitly state that their setup does not allow attributing alignment differences to a single factor (e.g., size, data amount) due to the off-the-shelf nature of the models tested.