Semantic Information Orthogonal to Visual Features Peaks in LateralOccipitotemporal Cortex

Using 7T fMRI data and a method to isolate semantic content from visual features, this study reveals that the lateral occipitotemporal cortex, particularly body-selective regions, encodes visually independent semantic information more robustly than ventral stream or early visual areas.

The Gist

The Big Question: Is Your Brain "Reading" or "Seeing"?

Imagine you are looking at a photo of a nurse chasing a man.

• Your eyes see pixels: skin tones, the shape of a uniform, the blur of motion.
• Your brain understands the story: "A nurse is chasing a man."

For a long time, scientists thought the part of your brain that processes images (the visual cortex) was mostly just a super-computer for analyzing shapes and colors. They thought the "story" part (the meaning) was handled by a different part of the brain entirely.

But this new study asks a tricky question: Does the image-processing part of your brain actually understand the story, or is it just really good at guessing the story based on how the picture looks?

The Experiment: The "Magic Eraser"

To find the answer, the researchers used a clever trick. They treated the brain like a radio and the images like songs.

1. The Setup: They showed 8 people thousands of photos while scanning their brains with a super-powerful MRI (7T fMRI). They also had computers describe these photos using advanced AI language models (like the tech behind Chatbots).
2. The Problem: The AI descriptions and the photos are naturally linked. If the AI says "a dog," the photo has a dog. So, if the brain reacts to the word "dog," is it reacting to the meaning of the word, or just the shape of the dog in the photo?
3. The Solution (The Magic Eraser): The researchers used a mathematical "eraser." They taught a computer to look at the photo and predict what the AI would say about it.
   • Step 1: The computer looks at the photo's pixels (visual features).
   • Step 2: It predicts the AI's description.
   • Step 3: It erases that prediction from the AI's description.
   • Result: What's left is the "pure meaning" that has nothing to do with what the picture looks like. It's the semantic "ghost" that remains after you remove the visual "body."

The Discovery: The "Body" Detective

They then asked: Which part of the brain lights up when we show it this "pure meaning" (the ghost) that has no visual shape attached?

They expected the answer to be the "Ventral Stream"—the classic "what is it?" pathway in the brain that handles faces, places, and objects.

The Surprise:
The strongest reaction didn't come from the face area or the place area. It came from the Lateral Occipitotemporal Cortex, specifically a region called EBA (Extrastriate Body Area).

• The EBA is usually known as the "Body Detector." It lights up when you see a human body.
• The Finding: Even after removing all the visual shapes of the body, the EBA still reacted strongly to the meaning of the body.
  • Analogy: Imagine you have a radio that only plays music when you show it a picture of a guitar. But this study found a radio that plays music when you tell it the concept of "guitar," even if you show it a picture of a piano. The EBA isn't just seeing the body; it's understanding the role and story of the body.

The "Negative" Proof: The Early Visual Cortex

To make sure their "Magic Eraser" actually worked, they looked at the very first part of the visual system (V1), which is like the camera sensor in your eye.

• The Result: When they showed this "pure meaning" (the ghost) to the camera sensor, the brain went negative. It actually reacted less than if they showed it nothing.
• Why this matters: This is like a quality control check. If the eraser had failed and left some visual "dirt" behind, the camera sensor would have lit up. The fact that it went dark proves the eraser worked perfectly. The signal they found in the EBA is genuinely about meaning, not about pixels.

The "Lateral vs. Ventral" Showdown

The study compared two main highways in the brain:

1. The Ventral Stream (The "Object" Highway): Good at faces (FFA) and places (PPA).
2. The Lateral Stream (The "Body/Action" Highway): Good at bodies (EBA).

The Verdict:
The "Object" highway (Ventral) was mostly just reacting to the visual shapes. Once you erased the shapes, the meaning didn't matter much.
The "Body" highway (Lateral/EBA) was different. Even without the shapes, it was still screaming, "I understand the story!"

• Analogy: Think of the Ventral stream as a photographer who cares about lighting and composition. Think of the Lateral stream (EBA) as a detective who cares about the motive and the relationship between people. Even if you hide the suspect's face (remove visual features), the detective still knows who they are based on the context.

Why Does This Matter?

This study changes how we think about the brain. It suggests that the part of your brain responsible for seeing bodies isn't just a camera; it's a social processor.

When you see a person, your brain isn't just calculating the curve of an arm; it's instantly processing the social meaning of that arm. Is it waving? Is it fighting? Is it comforting?

The study proves that meaning is baked into the visual system, specifically in the areas that handle bodies and social interaction. Your brain doesn't just see the world; it understands the story of the world, even when the visual details are stripped away.

Summary in One Sentence

By using a mathematical "eraser" to remove visual details from images, scientists discovered that the part of your brain that sees bodies is actually a master of understanding social stories, far more so than the parts of your brain that recognize faces or places.

Technical Summary

1. Problem Statement

While previous research has established that language model (LLM) embeddings can predict fMRI responses in the human visual cortex, a critical ambiguity remains: Does this alignment reflect truly visually-independent semantic content, or does it merely occur because LLMs are better at approximating complex visual features than traditional computer vision models?

Existing studies comparing language and vision models often fail to condition on low-level and mid-level visual features. Consequently, it is unclear whether the "semantic" variance explained by LLMs is genuinely distinct from visual processing or simply a superior representation of visual structure. The authors aim to resolve this by isolating the component of language model embeddings that is orthogonal (statistically independent) to a comprehensive hierarchy of visual features and determining which brain regions encode this residual semantic information.

2. Methodology

The study utilizes data from the Natural Scenes Dataset (NSD), a large-scale 7T fMRI dataset where 8 participants viewed tens of thousands of MS-COCO images.

A. Feature Extraction

• Visual Features ($X_{vis}$): Two complementary sets were extracted for every image:
  1. L1 (Low-level): Gabor filterbanks (orientation, scale), HSV color histograms, and Canny edge density.
  2. L2 (High-level): Deep features from a pre-trained VGG19 network (global average pooling output).
  • Note: In robustness checks, additional models (ResNet50, CLIP-visual, EfficientNet-B0) were included.
• Semantic Features ($B$): Text descriptions (COCO categories + captions) were encoded using BERT (and later GPT-2 and CLIP-text) to generate 768-dimensional embeddings.

B. Visual Residualisation (The Core Innovation)

To isolate semantic content independent of vision, the authors employed a cross-validated residualisation procedure:

1. Linear Projection: A Ridge regression model was trained to predict the BERT embeddings ($B$) from the visual features ($X_{vis}$).
2. Residual Calculation: The predicted visual component ($\hat{B}$) was subtracted from the original BERT embeddings to create residual embeddings ($\tilde{B} = B - \hat{B}$).
3. Orthogonality: By construction, $\tilde{B}$ is linearly orthogonal to the visual feature space. This ensures that any subsequent prediction of brain activity using $\tilde{B}$ cannot be attributed to visual features modeled by the pipeline.

C. Voxelwise Encoding & Analysis

• Prediction: Cross-validated Ridge regression was used to predict voxelwise fMRI responses using the residual embeddings ($\tilde{B}$).
• Metric: The primary metric was $R^2_{wiped}$, representing the variance in fMRI explained only by the visually-independent semantic component.
• Validation:
  • Negative Control: Early visual cortex (V1–V3) was expected to show negative $R^2_{wiped}$ because the residuals are orthogonal to visual structure.
  • Robustness: The pipeline was repeated across 6 conditions (3 Language Models: BERT, GPT-2, CLIP-text $\times$ 2 Visual Feature sets) to ensure architecture invariance.
  • Statistics: Group-level inference used random-effects t-tests and exact sign-flip permutation tests ($N=8$ subjects) to confirm significance without distributional assumptions.

3. Key Results

A. Lateral vs. Ventral Dissociation

The most significant finding is a clear dissociation between lateral and ventral streams:

• Lateral Occipitotemporal Cortex (LOTC): Specifically the Extrastriate Body Area (EBA) and the Lateral Stream, showed the strongest positive $R^2_{wiped}$.
  • Effect Size: EBA (Right Hemisphere) had $R^2_{wiped} \approx 0.039$ with a Hedges' $g \approx 2.90$.
  • Semantic Independence Ratio: Approximately 17–18% of the total explainable variance in EBA is purely semantic (orthogonal to vision), compared to only ~5% in place-selective regions (PPA) or the ventral stream.
• Ventral Stream: Regions like FFA (faces), PPA (places), and RSC showed significantly lower residual semantic encoding (3–4 times lower than EBA).

B. The "Negative Control" in Early Visual Cortex

• V1–V3: These regions exhibited robustly negative $R^2_{wiped}$ (e.g., $R^2 \approx -0.014$ in V1v).
• Significance: This confirms the validity of the residualisation method. Since the residuals are orthogonal to visual features, they "anti-predict" voxels that rely entirely on visual structure. This rules out the possibility that the method merely amplifies noise or fails to remove visual signals.

C. Architecture Invariance

The lateral-over-ventral dissociation held true across all 6 combinations of models:

• Language Models: GPT-2 (text-only), BERT (text with indirect visual grounding), and CLIP-text (text trained with direct image alignment).
• Visual Features: Broad ensembles (Gabor + VGG + ResNet + CLIP + EfficientNet) vs. hierarchical VGG19.
• Observation: GPT-2 produced the largest residuals (as it has no visual training), followed by BERT, then CLIP-text. Crucially, even with CLIP-text (which is heavily visually grounded), EBA still showed significant positive residual prediction, while early visual cortex remained negative.

D. Secondary Finding: VWFA-2

The left Visual Word Form Area 2 (VWFA-2) also showed a high semantic independence ratio (~17%), comparable to EBA, suggesting it retains significant non-visual semantic content, though this was specific to the left hemisphere.

4. Key Contributions

1. Methodological Novelty: Introduced a visual residualisation pipeline that explicitly separates language model variance from visual feature variance in voxelwise encoding models. This moves beyond simple model comparison to isolate unique semantic content.
2. Resolution of the EBA Debate: Challenges the view of EBA as a purely perceptual body detector. The results demonstrate that EBA carries substantial visually-independent semantic information (likely related to social roles, actions, and relational context) that is not captured by standard visual feature hierarchies.
3. Validation via Negative Control: The robust negative signal in early visual cortex serves as a mechanistic proof that the "wiped" signal is genuinely orthogonal to vision, strengthening confidence in the positive findings in higher cortex.
4. Architecture-Agnostic Discovery: Demonstrated that the concentration of independent semantic coding in the lateral stream is a fundamental property of the visual system, replicating across diverse LLMs and visual architectures.

5. Significance and Implications

• Reframing Visual Semantics: The study suggests that the human visual system does not just encode "what objects look like" but also encodes semantic structure orthogonal to visual features, particularly regarding bodies and social interactions.
• Functional Role of EBA: Supports the hypothesis that EBA is involved in higher-order social and relational processing (e.g., understanding dyadic interactions) rather than just shape/posture detection.
• Future Modeling: For computational neuroscience, this implies that feedforward visual models systematically underpredict activity in body-selective and social-perception regions. Future encoding models must incorporate residualised language embeddings to achieve higher prediction ceilings in these areas.
• Theoretical Boundary: The findings suggest a heterogenous organization of semantic coding in the occipital cortex, where the lateral stream retains more amodal semantic variance than the canonical ventral stream.

In summary, the paper provides compelling evidence that the Extrastriate Body Area (EBA) and Lateral Stream are the primary hubs in the human visual cortex for processing semantic information that is distinct from, and independent of, low-to-mid-level visual features.