🧠 neuroscience

Face and body representations converge along the visual hierarchy in models and cortex

This study demonstrates that both deep neural network models and the human visual cortex exhibit a progressive integration of face and body representations along the visual hierarchy, where mixed-selective units emerge to support unified person perception.

Original authors: van Dyck, L. E., Dobs, K.

Published 2026-02-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: van Dyck, L. E., Dobs, K.

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

The Big Question: Do We See People as "Heads + Bodies" or as "Whole People"?

Imagine you are walking down a busy street. You see a friend waving. Your brain instantly recognizes them. But how does it do that?

For a long time, scientists have debated a central question: Does our brain have two separate "files" for people—one for faces and one for bodies—or does it have a single "file" that combines them into one whole person?

Some scientists thought the brain was like a strictly organized library: one shelf for faces, a completely different shelf for bodies, and they never mixed. Others thought it was more like a smoothie: faces and bodies are blended together so thoroughly you can't tell where one ends and the other begins.

This new study, using both computer models and brain scans, suggests the answer is a bit of both. It's like a gradual blending process.

The Experiment: Training Digital Brains

The researchers didn't just look at human brains; they built "digital brains" (Deep Neural Networks) to see how they learned to recognize people. They trained these computers on millions of images, just like a child learns by seeing the world.

The Discovery: The "Hybrid" Workers
As the computer models got smarter, they developed three types of "neurons" (tiny processing units):

Face Specialists: Units that only cared about faces.
Body Specialists: Units that only cared about bodies.
The "Hybrids" (Mixed-Selectivity): Units that cared about both faces and bodies.

The Analogy: Think of a construction site.

At the bottom (early layers), you have workers who only lay bricks (faces) or only pour cement (bodies).
As you go up the building (later layers), you start seeing foremen who can manage both the bricklayers and the cement pourers. These "Hybrid" workers are the most common in the higher levels of the brain.

The Brain Scan Connection

The researchers then checked if this "Hybrid" pattern existed in real human brains using fMRI (a machine that takes pictures of brain activity).

The Finding:
They found that the parts of the brain that usually light up when we see faces or bodies are actually best explained by these Hybrid units.

Early Brain Areas (The Back): These areas are like the "specialist workers." They are very good at spotting just a face or just a body, but they don't mix them much yet.
Later Brain Areas (The Front): As you move toward the front of the brain, the signals start to merge. The brain begins to treat the face and body as a single package: "That is a person."

The Metaphor: Imagine a river.

At the source (the back of the brain), you have two distinct streams: one clear blue (faces) and one muddy brown (bodies).
As the river flows downstream (toward the front of the brain), the streams merge. By the time they reach the ocean (the front of the brain), you can't tell where the blue ended and the brown began; it's just one big, unified flow of "Person."

Why Do We Need "Hybrid" Units?

You might ask, "If we have specialists, why do we need hybrids?"

The study found that these Hybrid units are incredibly useful for flexibility.

If you need to recognize a specific face, the Face Specialist is the star.
If you need to recognize a specific person (who they are, regardless of whether you see their face or just their body), the Hybrid units are the stars.
If you need to guess what someone is doing (running, climbing), the brain uses a mix of everything.

The Analogy: Think of a Swiss Army Knife.

The "Face Specialist" is like the dedicated screwdriver. It's great for screws.
The "Body Specialist" is like the dedicated scissors. Great for cutting.
The "Hybrid Unit" is the main handle of the knife. It doesn't do one thing perfectly, but it holds everything together and allows you to switch tasks quickly. It lets the brain say, "That's not just a face, and not just a body; that's Bob running."

The Twist: It's Still "Part-Based"

Here is the most surprising part. The researchers thought these Hybrid units might be creating a magical, new "whole person" signal that is greater than the sum of its parts.

The Reality: They found that the Hybrid units are actually just adding the parts together.

The Analogy: Imagine you are listening to a duet. You have a singer (face) and a guitarist (body). The Hybrid unit is the sound mixing board. It doesn't create a new "super-sound"; it just plays the singer and the guitarist at the same time. It's a "part-based" mix, not a magical fusion.

The Takeaway

Our brains don't strictly separate faces and bodies, nor do they completely blend them into a mystery soup. Instead, they use a smart, progressive strategy:

Start Separate: Early on, the brain spots the parts (face vs. body).
Blend Gradually: As the information moves forward, the brain creates "Hybrid" workers that hold both pieces of information together.
Result: This allows us to be specialists when we need to be (recognizing a face) and generalists when we need to be (recognizing a whole person in action).

It's a perfect balance between having a specialized toolbox and a versatile Swiss Army Knife, all working together to help us navigate the social world.

1. Problem Statement

The perception of other people relies on visual cues from both faces and bodies. While neuroimaging has identified distinct face-selective (e.g., FFA, OFA) and body-selective (e.g., FBA, EBA) regions in the human visual cortex, their functional relationship remains debated. Three competing theories exist:

Strict Segregation: Faces and bodies are processed in entirely separate pathways.
Full Integration: They share a continuous representational space.
Partial/Progressive Integration: Representations are segregated in posterior regions but integrate as they move anteriorly along the visual hierarchy.

Furthermore, it is unclear whether whole-person representations are holistic (synergistic) or merely the sum of their parts (additive). This study aims to resolve these questions by combining Deep Neural Networks (DNNs) as computational models with large-scale fMRI data to test how face and body signals are segregated or integrated.

2. Methodology

A. Deep Neural Network (DNN) Analysis

Models: The authors analyzed multiple architectures, primarily focusing on AlexNet variants trained on Ecoset (supervised) and ImageNet (self-supervised via Barlow Twins). Additional models included VGG16.
Unit Classification: Instead of pooling spatial positions, every spatial position in convolutional layers was treated as a distinct unit. Units were classified into three mutually exclusive types based on responses to grayscale images of faces, bodies, scenes, and objects:
1. Face-selective: Prefer faces over all other categories.
2. Body-selective: Prefer bodies over all other categories.
3. Mixed-selective: Prefer both faces and bodies over other categories (but not necessarily equally).
Feature Visualization: Guided Grad-CAM was used to identify the visual features driving each unit type.
Lesioning Experiments: To test functional relevance, specific unit types were "lesioned" (activations set to zero) in a fixed DNN backbone. Linear classifiers were then trained on the penultimate layer (fc7) to perform three tasks:
- Face Recognition (VGGFace2)
- Person Recognition (Celeb-reID)
- Action Recognition (Stanford 40 Actions)
Integration Paradigm: Units were tested on four conditions: whole persons, isolated faces, isolated bodies, and backgrounds. Linear contrasts ( $\beta_F, \beta_B, \beta_{FB}$ ) were computed to determine if responses to whole persons were additive (sum of parts) or synergistic (super-additive).

B. fMRI Encoding Analysis

Dataset: The Natural Scenes Dataset (NSD), containing high-resolution (7T) fMRI data from 8 participants viewing thousands of natural images.
Region of Interest (ROI) Definition: Functional localizers identified face-selective (OFA, FFA, aTL-faces), body-selective (EBA, FBA, mTL-bodies), and overlap regions. Voxels selective for both were excluded from pure regions and grouped into a dedicated overlap region.
Encoding Models: Cross-validated voxel-wise encoding models (non-negative ridge regression) were fitted to predict fMRI activity from DNN unit activations.
Variance Partitioning: Commonality analysis was used to decompose the explained variance ( $R^2$ $R^{2}$ ) into:
- Unique variance explained by face units.
- Unique variance explained by body units.
- Shared variance (Face $\cap$ Body).
- Additional unique variance explained by mixed units.

3. Key Results

A. Emergence of Mixed Selectivity in DNNs

Prevalence: DNNs optimized for visual recognition develop distinct sets of face, body, and mixed-selective units.
Hierarchical Progression: The proportion of mixed-selective units increases systematically from early to late layers (Spearman $r > 0.9$ ). Mixed units emerge in intermediate layers, suggesting integration begins at mid-level feature processing.
Generalization: These selectivity profiles generalize to novel images (independent datasets), confirming they rely on learned features rather than dataset artifacts.
Feature Specificity: Saliency maps show face units respond to facial features, body units to body contours (often including heads), and mixed units to both.

B. Mapping to Human Visual Cortex

Predictive Power: Mixed-selective units were the strongest predictors of fMRI activity in both face-selective and body-selective regions, outperforming purely selective units in most areas (except early regions like OFA and EBA).
Variance Explanation:
- Face and body units contributed unique variance to their corresponding regions (double dissociation).
- However, the majority of explained variance was shared between face and body units.
- Mixed units explained little additional unique variance beyond what face and body units already shared, suggesting they encode a combination of existing signals.
Posterior-to-Anterior Gradient: The proportion of integrated variance (shared variance + mixed unit variance) increased systematically from posterior (early) to anterior (late) cortical regions. This supports a model of progressive convergence.

C. Functional Role and Integration Mechanism

Task Relevance: Lesioning experiments revealed that mixed units carry multifaceted information:
- Face Recognition: Relies primarily on face units; mixed units provide secondary support.
- Person Recognition: Relies primarily on body units; mixed units provide secondary support.
- Action Recognition: Relies on a distributed mix of all unit types.
Additive vs. Synergistic: Contrary to the hypothesis of synergistic integration, the study found that units (including mixed ones) process whole persons in a part-based, additive manner. Responses to whole persons were generally sub-additive (less than the sum of face + body responses) or strictly additive. No synergistic integration (super-additivity) was observed.

4. Key Contributions

Computational Resolution of a Debate: The study provides strong evidence against strict segregation and full integration, supporting a partially integrated model where representations converge progressively along the visual hierarchy.
DNN-Brain Alignment: It demonstrates that task-optimized DNNs naturally develop mixed-selective units that map onto human cortical organization, validating DNNs as models for high-level visual processing.
Nature of Whole-Person Representation: It challenges the notion of holistic, synergistic integration for whole persons. Instead, it suggests that "whole-person" representations in both models and cortex are constructed via additive, part-based processing of face and body cues.
Functional Flexibility: It establishes that mixed-selective units serve a flexible role, allowing the system to read out information for diverse tasks (identity, action) without requiring dedicated "whole-person" neurons.

5. Significance

This work fundamentally shifts the understanding of person perception from a binary "segregated vs. integrated" debate to a nuanced view of progressive convergence. It suggests that the visual system balances specialization (pure face/body units for specific tasks) with flexibility (mixed units for general person perception).

The finding that integration is additive rather than synergistic has profound implications for theories of object recognition, suggesting that the brain may not need dedicated neural populations for complex combinations (like whole persons) but can achieve this through the linear combination of part-based features. This offers a computationally efficient mechanism for recognizing complex social stimuli in natural environments.