Shared representations in brains and models reveal a two-route cortical organization during scene perception

Problem
The human brain transforms visual inputs into high-dimensional cortical representations to support diverse cognitive goals. While it is established that sensory signals are converted into structured patterns of cortical activity across multiple stages (from early visual cortex to downstream regions), characterizing how this information is organized and routed across the human brain remains a central challenge. Specifically, there is a need to identify which regions encode specific aspects of sensory input, map the networks that route this information, and distinguish between visually grounded features and more abstract semantic representations. Traditional models often rely on the "ventral what / dorsal where" dichotomy, but recent proposals suggest more complex, fractionated pathways for scene context and social perception that require empirical validation using naturalistic stimuli.

Methodology
The authors applied a unified framework to the Natural Scenes Dataset (NSD), a high-resolution 7T fMRI dataset where eight participants viewed thousands of natural scenes while performing a memory recognition task. The analysis pipeline integrated four complementary approaches:

1. Feature Extraction: Cortical surfaces were parcellated into 180 regions per hemisphere (HCP-MMP atlas). Single-trial fMRI responses were aggregated within parcels to create vector representations. Concurrently, layer-wise activations were extracted from 37 pretrained deep neural networks (DNNs), including 17 Vision Transformer (ViT) models and 14 Transformer-based language models (LLMs), processing both the images and their text captions.
2. Representational Alignment (RSA): The authors used Representational Similarity Analysis to quantify correspondence in two ways:
   • Inter-subject RSA (IS-RSA): Correlating parcel-wise Representational Dissimilarity Matrices (RDMs) across participants to estimate shared representational geometry.
   • Brain-Model RSA: Correlating parcel RDMs with model-layer RDMs to quantify alignment with vision and language feature spaces.
3. Representational Connectivity: IS-RSA was repurposed as a measure of connectivity between pairs of parcels to construct a cortical network based on the similarity of how regions organize the stimulus set, rather than anatomical proximity.
4. Dimensionality Decomposition: Kernel Multi-view Canonical Correlation Analysis (KMCCA) was applied to decompose shared geometry within identified hubs into latent dimensions, relating these dimensions to coarse stimulus properties (e.g., scene layout, biological presence).

To ensure robustness, the pipeline was replicated on two independent datasets (BOLD5000 and THINGS-fMRI) with different stimulus sets and task demands.

Key Contributions

• Unified Framework: The study introduces a methodology that simultaneously localizes shared information, characterizes its routing as a distributed network, and identifies the specific stimulus features driving organization, linking these to hierarchical features in DNNs.
• Two-Route Organization: The analysis reveals a dissociable two-route cortical organization during scene perception, refining classical visual-stream models.
• Modality-Specific Alignment: The work demonstrates distinct alignment patterns between brain regions and AI models: vision models align with both processing routes, whereas language models align primarily with the lateral route.
• Stimulus-Driven Geometry: The study quantifies how specific stimulus properties (scene layout vs. biological agents) drive the shared geometry in different network hubs.

Results

1. Shared Geometry and Hubs: IS-RSA identified significant alignment across the occipital cortex extending into higher-order association areas. Three anatomically clustered hubs showed the strongest shared geometry:
   • Early Visual Cortex (V1-V4): Encoding low-level visual features.
   • Ventral Hub: Located in ventromedial temporal cortex (VMV1-3, PHA1-3), overlapping scene-selective regions like the Parahippocampal Place Area (PPA).
   • Lateral Occipitotemporal Cortex (LOTC) Hub: Including the motion-selective MT+ complex and posterior temporoparietal junction (TPOJ), regions implicated in biological motion and social perception.
2. Two Distinct Processing Routes: Representational connectivity analysis revealed that the Early Visual hub bifurcates into two dominant routes:
   • Medial-Ventral Route: Progresses through ventral occipitotemporal cortex to the Ventral hub. This route is specialized for scene layout and environmental context. It aligns with vision models across multiple depths but shows little to no alignment with language models.
   • Lateral Route: Progresses through lateral occipital cortex to the LOTC hub. This route is selectively tuned to animate and social content. It aligns with later stages of vision models and, crucially, shows strong alignment with language models.
3. Stimulus Drivers: KMCCA decomposition revealed that the shared geometry in the Ventral hub varies along a scene-to-object continuum. In contrast, the LOTC hub's geometry is dominated by a dimension separating images with biological agents (people/animals) from those without. Removing biological content significantly reduced inter-subject alignment in the LOTC hub and weakened the lateral route's connectivity.
4. Model Correspondence: Vision models captured the shared geometry of both routes, with a posterior-to-anterior gradient in alignment depth (shallow layers for Early Visual, deep layers for LOTC). Language models, however, showed alignment concentrated almost exclusively in the LOTC hub, driven by token-level semantic content ("bag-of-words" organization) rather than a hierarchical depth gradient.
5. Generalizability: Replication on BOLD5000 (rich scenes) confirmed the two-route topology, though with reduced magnitude due to lower signal reliability. In THINGS-fMRI (object-centric, minimal scene layout), the lateral route and LOTC hub were not prominent, confirming that the two-route organization is dependent on the presence of rich scene and social content in the stimulus set.

Significance
The paper claims to refine classical visual-stream models by characterizing scene perception as a distributed cortical network with separable representational routes for context and animate content. It provides evidence supporting proposals of a ventromedial pathway for scene context and a lateral "third visual pathway" for social perception. By integrating alignment, connectivity, and dimensionality decomposition, the study offers a practical route for leveraging large-scale data to move beyond asking if systems encode information similarly, toward understanding what information is shared and how it is distributed. The authors note that these findings are context-dependent, varying with stimulus sampling and task demands, and that the framework identifies representational organization rather than the precise mechanistic implementation. The work establishes a benchmark for future research into how distributed cortical systems support perception in dynamic environments.