🧠 neuroscience

Language-aligned models and structured scene descriptions reveal sensitivity to compositional scene structure in the high-level visual cortex

Using 7T fMRI data and language-aligned models, this study demonstrates that the high-level visual cortex is sensitive to the compositional structure of natural scenes rather than just object co-occurrence, suggesting that language supervision can help artificial vision models develop similar structured representations.

Original authors: Rajaei, K., Afshar, A., Cichy, R. M., Soltanian-Zadeh, H.

Published 2026-07-28

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Rajaei, K., Afshar, A., Cichy, R. M., Soltanian-Zadeh, H.

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine your brain is a massive, bustling library. For a long time, scientists thought the "visual section" of this library was just a giant filing cabinet for individual objects. If you saw a dog, a car, or a tree, a specific shelf would light up to say, "Hey, we have a dog here!" But what happens when the dog is chasing the car, or the tree is blocking the car? Is the brain just checking off a list of items, or is it actually reading the story of how they fit together? This question sits at the heart of a field called cognitive neuroscience, where researchers try to decode the secret language of our thoughts and senses. They use powerful tools like fMRI (functional Magnetic Resonance Imaging), which acts like a high-tech camera that can see which parts of the brain are "talking" when you look at something. The big mystery is whether our brain's visual center is smart enough to understand the relationships between things, or if it's just a list-maker that only cares about what objects are present.

A team of researchers decided to crack this code by treating the brain like a detective and language like a magnifying glass. They used data from eight people who had their brains scanned while looking at 10,000 different natural scenes. To test the brain's ability to understand "stories" rather than just "lists," they created a clever experiment. For every picture, they used an AI to write a full, grammatical sentence describing exactly what was happening (e.g., "A person is chasing a dog in a park"). Then, they took that same sentence and scrambled it into a random pile of words (e.g., "park, dog, person, chasing, in, a"). Both versions had the exact same words, but only the first one told a coherent story with a clear structure.

The results were a fascinating reveal. When the researchers tried to predict what the brain was doing based on these descriptions, the scrambled "bag of words" still predicted brain activity, but it was significantly less accurate than the full, structured sentences. The structured sentences provided a much better match for the brain's activity, especially in the high-level areas responsible for recognizing faces, places, and bodies. It turns out the brain isn't just a list-maker; it's a storyteller. It cares deeply about how the pieces fit together. The researchers also tested this with computer models. They found that AI models trained to understand both images and language were much better at guessing brain activity than models that only looked at pictures. This suggests that language helps the brain (and perhaps AI) organize the visual world into meaningful connections, not just a jumble of objects.

To make sure this wasn't just about the sentences sounding "fluent" or grammatically correct, the team tried a second trick. They took the key words from the original story and asked an AI to write a new grammatical sentence using only those words. Even though this new sentence was grammatically perfect, it didn't predict the brain's reaction as well as the original, image-specific story. This proved that the brain isn't just happy to hear any well-formed sentence; it specifically craves the unique, structured relationships that exist in the actual scene. The study suggests that the high-level visual cortex is sensitive to the "compositional structure" of a scene—the specific way agents, actions, and spaces are organized. It's not enough to know a dog and a person are there; the brain needs to know who is chasing whom to truly "see" the scene.

Technical Summary

Problem and Motivation
Natural scenes are defined not merely by the objects they contain, but by the structured relations among those entities, including spatial layout, semantic associations, social interactions, and actions. While the visual system rapidly extracts such structural information to interpret scene meaning, it remains unclear whether high-level visual cortex representations encode this compositional structure or primarily reflect the co-occurrence of individual entities. Previous work has established that language-aligned vision models predict brain responses better than vision-only models, but the basis of this advantage is debated: it may stem from richer semantic labels or from the ability to capture structured relational organization. This study aims to dissociate sensitivity to structured scene organization from sensitivity to the mere presence of scene elements.

Methodology
The authors utilized 7T fMRI data from the Natural Scenes Dataset (NSD), comprising responses from eight participants viewing 10,000 unique natural images. The study employed two complementary approaches within an encoding framework:

Linguistic Manipulation: For each image, a generative vision-language model (Qwen 2.5-VL) produced a narrative description. The authors constructed multiple linguistic feature spaces to systematically manipulate structure while controlling for lexical content:
- Full Narrative: Intact, coherent descriptions.
- Bag-of-Words: The same words as the narrative but in randomized order, disrupting syntactic and relational structure.
- Content Words: Nouns, verbs, and adjectives in random order (excluding function words).
- Isolated Categories: Models trained on nouns, verbs, or adjectives alone.
- Regenerated Descriptions: A control where function words were removed from the original narrative, and a language model regenerated a grammatical sentence using only the original content words. This tested whether the narrative advantage was due to generic grammaticality or image-specific relational organization.
- Text embeddings were derived using CLIP (ViT-L/14) and RoBERTa encoders.
Computational Model Comparison: The authors compared neural predictivity between a language-aligned vision model (SigLIP2) and a self-supervised vision-only model (DINOv3). Both models shared similar architectures (Vision Transformer Large) and training scales but differed in their learning objectives (image-text contrastive loss vs. self-supervised visual learning).

Encoding and Analysis
Voxelwise ridge regression models were fitted to predict fMRI responses from the feature spaces. The study utilized variance partitioning to compute unique explained variance ( $R^2_{unique}$ ). This metric isolated the contribution of a specific feature set (e.g., narrative structure) by measuring the drop in prediction performance when that set was removed from a joint model containing both the target set and a baseline (e.g., bag-of-words). Statistical significance was assessed via non-parametric bootstrap resampling and FDR correction.

Key Results

Narrative Structure Outperforms Lexical Controls: Full narrative descriptions significantly predicted cortical responses better than bag-of-words and content-word controls across high-level visual cortex, including category-selective regions (EBA, FFA, PPA, OPA, RSC). This advantage was minimal in early visual cortex.
Specificity to Compositional Structure: The unique variance explained by narratives over bag-of-words was not driven by function words or generic fluency. The regenerated-description control showed that while grammatical sentences reconstructed from content words captured some variance, they failed to match the predictive power of the original image-specific narratives. This indicates the brain encodes the specific relational organization of the scene, not just fluent sentence forms.
Lexical Contributions: Nouns explained the most variance, followed by verbs and adjectives. However, unordered combinations of multiple lexical classes captured more information than any single category alone.
Convergence with Language-Aligned Models: SigLIP2 (language-aligned) outperformed DINOv3 (vision-only) in high-level category-selective regions. The spatial distribution of this advantage closely mirrored the distribution of the narrative advantage over bag-of-words. A strong positive correlation ( $r = 0.66$ ) was found between the unique variance explained by structured descriptions and the unique variance explained by the language-aligned model across voxels.
Stimulus Drivers: Images driving the unique variance for both structured descriptions and language-aligned models were characterized by high scores in compositional structure, semantic relations, and social interaction. These were not simply "visually rich" images but those with complex, organized relations among agents, objects, and actions.

Significance and Claims
The paper claims that the high-level visual cortex represents structured scene information that goes beyond simple entity co-occurrence or lexical content alone. The findings suggest that the visual system encodes the relational organization of scene elements (agents, actions, spatial layout) into a coherent whole.

Furthermore, the study posits that the superior performance of language-aligned vision models in predicting brain activity is not merely an artifact of broader lexical coverage but reflects their capacity to capture structured relational information similar to that represented in the human visual cortex. The convergence between the "narrative advantage" and the "language-aligned model advantage" supports the view that language supervision helps induce representations in artificial models that align with the structured relational coding of the biological visual system. The authors conclude that natural language serves as a useful probe for isolating relational information encoded by the visual cortex, rather than acting as a substitute for vision.

Technical Summary

More like this