What Makes Linguistic Representations Good Models of High-Level Visual Perception in the Human Brain?

Problem Statement
Recent studies have demonstrated that image descriptions (captions) represented by large language models (LLMs) can predict neural responses in high-level visual areas of the human brain, challenging the notion that visual perception relies solely on strictly visual features. However, the specific factors driving this brain predictivity remain unclear. It is uncertain whether the performance is driven by the semantic content of the captions, the fluency and detail of the text, or the specific language model architecture used to generate the embeddings. Furthermore, previous work has relied exclusively on human-annotated captions from the MS COCO dataset, which are often short and provided by crowdworkers, potentially limiting the quality of the linguistic representations. This study seeks to systematically investigate how variations in caption content (generated by different vision-language models) and variations in the language models used to embed these captions influence brain predictivity and behavioral alignment.

Methodology
The authors utilized the Natural Scenes Dataset (NSD), which contains high-resolution fMRI responses from eight participants viewing 906 naturalistic images from the MS COCO dataset. No image descriptions were presented to participants during the fMRI scans.

1. Caption Generation: For each of the 906 images, the authors generated multiple captions using five open-source Vision-Language Models (VLMs): LLaVA OneVision, Phi-4, Pixtral, Qwen2.5-VL, and Molmo. These were compared against the original human-annotated MS COCO captions. The generated captions varied systematically in length, fluency (measured by perplexity), visualness (based on Lancaster Sensorimotor Norms), and lexical density.
2. Embedding Generation: Each caption was embedded using five distinct open-source, text-only Language Models (LMs) to ensure no visual information "contaminated" the linguistic representations. The LMs included:
   • An early masked language model (BERT).
   • Two autoregressive models (GPT-2, Llama3.1).
   • Two recent text embedders fine-tuned on semantic tasks (Qwen3 Embedding, KaLM Embedding).
   • Embeddings were extracted from all layers of these models.
3. Evaluation Metrics:
   • Brain Encoding: Voxel-wise regularized linear regressions (Ridge regression) were trained to predict neural activations in three functionally localized Regions of Interest (ROIs): face-selective, body-selective, and place-selective areas. Performance was measured by Pearson correlation between predicted and observed activations.
   • Brain Alignment: Representational Similarity Analysis (RSA) was used to compute the Spearman correlation between Representational Dissimilarity Matrices (RDMs) derived from caption embeddings and those derived from neural activations.
   • Behavioral Alignment: RSA was also applied to compare caption embeddings against human image-similarity judgments collected from the same NSD participants.
   • Intrinsic Dimensionality: The authors used the Generalised Ratios Intrinsic Dimension Estimator (GRIDE) to analyze information compression across model layers.

Key Results

• Machine-Generated vs. Human Annotations: Machine-generated captions significantly predicted neural activations, often outperforming the human-annotated MS COCO captions used in prior work. Specifically, captions generated by Qwen2.5-VL, Pixtral, and Molmo showed the highest predictivity. The authors attribute the lower performance of MS COCO captions partly to their lower fluency and brevity, noting that averaging multiple MS COCO captions improved performance to match single machine-generated captions.
• Impact of Language Models: Text embedders (Qwen3 and KaLM) consistently outperformed autoregressive LMs (GPT-2, Llama3) and BERT in both brain predictivity and behavioral alignment. This suggests that models explicitly fine-tuned for semantic representation capture brain-relevant information more effectively than those trained primarily for next-token prediction.
• Layer-wise Analysis: Both brain predictivity and behavioral alignment peaked at intermediate network depths, shortly after a phase of high intrinsic dimensionality. This high-dimensional phase is theorized to mark the emergence of abstract syntactic and semantic structure.
• Dissociation in ROIs: A "double dissociation" was observed in layer-wise alignment. Alignment with face- and body-selective ROIs peaked in mid-layers and declined in later layers, whereas alignment with place-selective ROIs increased dramatically in the deepest layers. This suggests that different linguistic information is required to model similarity patterns in these distinct visual areas.
• Behavioral Superiority: Caption embeddings, particularly those from text embedders using machine-generated captions, achieved higher alignment with human image-similarity judgments than image features extracted from state-of-the-art vision models (ResNet-50 and ViT).

Significance and Claims
The paper claims to establish that both the content of image captions and the specific language model used to represent them are critical determinants of brain- and behavior-modelling performance. The authors demonstrate that caption embeddings are a useful tool for studying high-level visual perception, capable of predicting neural activity and human similarity judgments with high accuracy.

Crucially, the study suggests that the brain predictivity of linguistic representations is not merely a byproduct of visual information leakage but is driven by the semantic properties of the text. The finding that text embedders outperform autoregressive models indicates that the ability to represent sentence-level meaning is paramount for modeling visual perception. Furthermore, the analysis of intrinsic dimensionality provides evidence that the emergence of abstract linguistic structure (signaled by a peak in intrinsic dimensionality) is the specific phase where representations become most predictive of human brain activity. Finally, the study offers the first analysis of NSD similarity judgments, revealing that linguistic representations can model human-perceived similarity even better than visual features, highlighting the deep interplay between language and visual perception.