Encoding models uncover fine-grained feature selectivity for bodies, hands and tools

By combining densely sampled fMRI data with artificial neural network-based encoding models, this study reveals that category-selective areas in the occipitotemporal cortex exhibit fine-grained, distinct feature sensitivities for bodies, hands, and tools that go beyond broad categorical tuning.

The Gist

Imagine your brain's visual system as a massive, bustling library. For a long time, scientists thought this library had a few big, general sections: a "Faces" section, a "Body" section, and a "Tools" section. If you walked into the "Body" section, you'd expect to see books about bodies, and that was it.

But this new study suggests the library is actually much more complex. It's not just one big room for "Bodies"; it's a collection of tiny, specialized study rooms, each with a very specific job. Some rooms are for whole bodies, some are for just hands, and some are for tools. Even more surprisingly, two rooms that both seem to be about "hands" might actually be reading different chapters of the same book.

Here is how the researchers figured this out, using a mix of brain scans and computer magic.

The Experiment: A Brain Scan "Pop Quiz"

The researchers put three volunteers in an MRI machine (a giant camera that takes pictures of the brain's activity). They showed them hundreds of pictures:

• Whole people
• Just hands
• Tools (like hammers or scissors)
• Other objects (like cups or buildings)

They asked the volunteers to pay attention to specific things, like spotting a bug, while their brains lit up in response to the images.

The "Virtual Brain" Trick

Here is the clever part. Instead of just looking at the brain scans and guessing what the brain was thinking, the researchers built AI "Virtual Twins" of these brain areas.

Think of it like this: They took the brain's reaction to the 200 pictures they showed, and they taught a computer program (a neural network) to mimic that reaction. Once the computer learned how the brain reacted, they didn't stop there. They let the computer "look" at millions of other images from the internet (like a massive digital photo album) that the humans never saw.

This allowed them to ask: "If this specific part of the brain saw a picture of a hammer, would it light up? What about a broom? Or a cat?"

The Big Discoveries

1. The "Hand" vs. "Tool" Confusion
The study found that the brain has distinct areas for hands and tools, but they are neighbors.

• The Left-Handed Specialist: One area (in the left side of the brain) loves hands, but it gets really excited when those hands are holding tools. It's like a mechanic who loves hands, but only when they are fixing a car.
• The Right-Handed Specialist: Another area (on the right side) also loves hands, but it prefers seeing hands as part of a whole body or in social contexts. It's more like an artist who loves drawing hands in portraits.

2. The "Tool" vs. "Object" Split
They found two different areas that respond to tools, but they see them differently:

• The "Action" Zone (Lateral): This area is like a workshop. It lights up for things you can grab and use—hammers, screwdrivers, scissors. It cares about how you interact with the object.
• The "Surface" Zone (Ventral): This area is more like a museum display case. It responds to tools, but it also lights up for big, non-graspable objects like streetlights or buildings. It seems to care more about the shape and material (like metal or wood) rather than the action.

3. The "Body" Nuance
Even for whole bodies, the brain isn't uniform. The left side of the brain seems to notice the shape and posture of bodies (and sometimes objects that look like bodies, like a tennis racket), while the right side is more sensitive to textures like fabric or wood.

Why This Matters

Think of the brain's visual system not as a set of rigid folders, but as a team of specialists.

• If you hand a hammer to the "Action Zone," it says, "I see a tool I can use!"
• If you hand the same hammer to the "Surface Zone," it says, "I see a shiny, metal object."
• If you show a hand holding that hammer to the "Left Hand Specialist," it says, "I see a hand doing work!"

The Takeaway:
This study proves that our brains are incredibly precise. Even within a category we think is simple (like "hands" or "tools"), our brain has carved out tiny, specialized neighborhoods. Each neighborhood has a slightly different job, focusing on different details like action, shape, or texture.

By using AI to act as a "translator" for these brain signals, the researchers could uncover these hidden layers of detail that we couldn't see just by looking at the raw brain scans. It's like realizing that a choir isn't just singing "music," but that every single singer is hitting a specific, unique note that creates the whole symphony.

Technical Summary

1. Problem Statement

While the ventral temporal cortex (VTC) and lateral occipitotemporal cortex (LOTC) are known to contain category-selective areas (e.g., for faces, bodies, tools), traditional neuroimaging often treats these regions as homogeneous units with broad tuning. However, emerging evidence suggests a finer-grained organization where:

• Anatomically close or overlapping areas (e.g., ventral vs. lateral OTC) may perform distinct computational roles.
• Hemispheric lateralization (left vs. right) influences feature sensitivity (e.g., hand vs. body selectivity).
• Current methods often lack the resolution to dissociate these sub-areas or to identify the specific visual features driving selectivity beyond simple category labels.

The study aims to:

1. Test the functional selectivity of body-, hand-, and tool-selective areas at the individual image level.
2. Use Artificial Neural Network (ANN)-based encoding models to uncover the specific feature representations that distinguish areas selective for the same category but located in different anatomical pathways (ventral vs. lateral) or hemispheres.

2. Methodology

Participants and Data Acquisition

• Subjects: Three right-handed participants.
• Sessions: Six scanning sessions per participant (approx. 9 hours of fMRI data total per person).
• Scanner: 3T Siemens Prisma with a 64-channel head coil.
• Resolution: High-resolution functional data (2x2x2 mm voxels) analyzed in native space to avoid smoothing-induced overlap between adjacent clusters.

Stimuli and Experimental Design

• Localizer (Block Design): Used to define Regions of Interest (ROIs). Included 6 categories: whole bodies, hands, tools, manipulable objects, non-manipulable objects, and chairs.
• Main Experiment (Rapid Event-Related): 200 unique images across 5 categories:
  • Whole bodies (25)
  • Hands (25)
  • Tools (50)
  • Manipulable objects (50)
  • Non-manipulable objects (50)
• Procedure: Images presented for 500ms with a 3500ms ISI. Participants performed a catch-trial detection task.

Region of Interest (ROI) Definition

ROIs were defined using independent localizer contrasts to ensure non-overlapping voxels:

• Body-selective: Whole bodies > (Hands + Tools)
• Hand-selective: Hands > (Whole bodies + Tools)
• Tool-selective: Tools > All other categories
• Locations: Bilateral Ventral OTC (VOTC) and Lateral OTC (LOTC).

Encoding Model Framework

• Feature Extraction: Images were processed through a ResNet-50 (pre-trained on ImageNet). The final global average pooling layer (2048-dimensional vector) was selected as the optimal feature layer based on correspondence with fMRI data.
• Model Training: Ridge regression was used to learn a linear mapping from ResNet-50 features to the mean fMRI BOLD response of each ROI.
• Validation: 10-fold cross-validation was employed.
• Large-Scale Screening: Trained models were tested on massive external datasets (ImageNet and EcoSet, totaling >2.5 million images) to identify the visual features driving the highest predicted activations.
• Analysis Techniques:
  • Clustering: Unsupervised k-means clustering (cosine distance) on the top 0.1% of predicted images to identify consistent visual themes.
  • Saliency Mapping: Occlusion-based mapping (RISE algorithm) to identify which image regions drove predictions.
  • Pairwise Contrasts: Comparing prediction scores between models (e.g., Left vs. Right hemisphere, Ventral vs. Lateral) to isolate differential feature sensitivities.

3. Key Results

A. Robust Image-Level Selectivity

• Encoding models successfully predicted neural responses for all ROIs, achieving high correlation coefficients ($r$) relative to the noise ceiling (e.g., LOTC-body right: $r=0.67$).
• Selectivity Metrics: All ROIs showed significant $d'$ values and high Area Under the Curve (AUC) scores (0.69–0.97), confirming that these areas are tuned to specific categories even at the single-image level.
• Top-N Analysis: The top 25 most activating images for each model were predominantly from the preferred category (e.g., 72% bodies for body-selective areas).

B. Fine-Grained Feature Dissociations

The study revealed that areas labeled with the same category selectivity encode distinct features based on location:

1. Hand-Selective Areas (Hemispheric Differences):

   • Left LOTC (Hand): Strongly tuned to handheld manipulable objects (tools like hammers, markers) and complex scenes involving interactions.
   • Right LOTC (Hand): Preferentially activated by whole body forms and richly textured backgrounds, showing less sensitivity to tools.
   • VOTC (Left Hand): Sensitive to single elongated objects (e.g., brooms) against simple backgrounds.

2. Tool-Selective Areas (Ventral vs. Lateral):

   • Left LOTC (Tool): Highly selective for canonical tools and highly manipulable objects (scissors, pliers, nails), often depicted with hands.
   • Left VOTC (Tool): Showed weaker selectivity for tools specifically. Instead, it responded broadly to large, non-manipulable inanimate objects (streetlights, buildings, vehicles) and squared objects. This suggests a shift from "tool identity" to "general inanimate object properties" in the ventral stream.

3. Body-Selective Areas:

   • Differences were subtler but present. Left LOTC body areas responded to inanimate objects with metallic textures (e.g., tennis rackets), while Right LOTC body areas responded to rectilinear textures and fabrics.

C. Feature Sensitivity via Saliency and Clustering

• Saliency Maps: Confirmed that body models focused on body parts (arms/shoulders), hand models focused strictly on hands, and tool models focused on graspable functional objects.
• Clustering: The top-ranked images for lateral tool areas consistently featured grasping interactions and metallic/elongated shapes, whereas ventral tool areas clustered around large, static inanimate structures.

4. Key Contributions

1. Validation of Fine-Grained Organization: The study provides empirical evidence that "category-selective" areas are not monolithic. Even within a single category (e.g., tools), there are distinct sub-clusters with different computational goals (action-related vs. surface/shape properties).
2. Methodological Advancement: Demonstrates the power of ANN-based encoding models combined with large-scale image screening (ImageNet/EcoSet) to move beyond "word models" (simple category labels) to "feature models" that reveal the underlying representational geometry of the brain.
3. Hemispheric and Pathway Dissociation:
   • Confirms the Left Hemisphere dominance for hand-tool interactions and action-related properties.
   • Confirms the Right Hemisphere dominance for body form processing.
   • Validates the Ventral-Lateral Gradient: Lateral OTC supports action-related properties (manipulability), while Ventral OTC supports general object recognition and surface properties.
4. Refinement of the "Tool" Concept: Challenges the binary view of tool selectivity, showing that ventral tool areas may actually represent a broader class of inanimate objects, while lateral areas are the true "tool" specialists.

5. Significance

This research bridges the gap between broad category selectivity and fine-grained feature representation in the human visual cortex. By showing that encoding models can dissociate overlapping functional areas based on their specific feature sensitivities, the study:

• Offers a new framework for understanding how the visual cortex organizes information not just by "what" an object is, but by "how" it is used (action) and its physical properties (shape/material).
• Suggests that the computational architecture of the OTC is driven by distinct goals: action planning (lateral/left) vs. object recognition/surface analysis (ventral/right).
• Provides a robust methodology for future studies to map the functional topography of the brain without relying solely on predefined category labels, potentially revealing new, previously undetected functional clusters.