Hyperface: a naturalistic fMRI dataset for investigating human face processing

The paper introduces "Hyperface," a high-quality, publicly available naturalistic fMRI dataset featuring 707 unique face video clips with systematic variations in identity and expression, designed to overcome the limitations of static stimuli and enable robust investigation of human face processing under ecologically valid conditions alongside benchmarking computational models.

The Gist

Imagine you've spent years studying how people recognize faces, but you've only ever done it in a sterile, quiet laboratory. You show your subjects a single, frozen photo of a person, ask them to identify the gender, and move on. It's like trying to understand how a chef cooks a complex meal by only looking at a single, still photograph of a raw potato. You miss the sizzling, the chopping, the steam, and the way the ingredients interact in real-time.

That's exactly the problem the authors of this paper, Hyperface, are trying to solve. They've created a massive new "recipe" for studying the human brain that is much more like real life.

Here is the story of Hyperface, explained simply:

1. The Problem: The "Frozen Photo" Trap

For decades, scientists have studied face recognition using static images. It's easy to control, but it's not how we actually see people. In the real world, faces move, talk, change expressions, and turn their heads. They are dynamic, messy, and full of life. The old way of studying faces was like trying to learn how to drive a car by only looking at a picture of a steering wheel.

2. The Solution: A "Face Video Buffet"

The researchers built a new dataset called Hyperface. Think of this as a massive, high-definition buffet of 707 short video clips (each 4 seconds long).

• The Ingredients: These aren't just random clips. They are carefully selected from YouTube interviews to include a huge variety of people: different ages, genders, ethnicities, and expressions. Some are smiling, some are serious, some are looking left, some right.
• The Guests: They invited 21 people into an MRI machine (a giant, noisy camera that takes pictures of the brain).
• The Meal: Instead of showing them frozen photos, they played these video clips on a screen while the participants' brains were being scanned. The participants were told to pay attention to who the person was, just like you would when meeting someone at a party.

3. The "Taste Test" (Behavioral Data)

To make sure the videos were actually useful, the researchers didn't just rely on the brain scans. They also asked hundreds of people online (via Amazon Mechanical Turk) to do two things:

1. Rate the ingredients: "Is this person male or female? Old or young? Happy or angry?" This created a detailed "flavor profile" for every single video clip.
2. Group the dishes: They asked people to arrange the clips in a circle based on how similar the faces looked. This helped them understand how humans naturally group faces together in their minds.

4. The "Super-Scanner" (The MRI Setup)

The brain scans were taken with a very high-quality 3T MRI scanner.

• The Headcase: To keep the brain still (because even a tiny head wiggle blurs the picture), the participants wore a custom-fitted helmet called a "CaseForge." It's like a cozy, custom-molded pillow that keeps your head from bobbing around.
• The Quality Check: The researchers checked the data to make sure it was clean. They found that the participants barely moved, the signal was strong and clear (like a high-definition TV signal), and everyone's brains reacted in a very similar way to the videos. This proves the data is reliable.

5. The "Grand Hotel" Connection

This isn't just a one-off experiment. The same 21 people also watched a movie called The Grand Budapest Hotel in a separate session. This is like having the same actors perform in both a short, focused sketch and a full-length play. Because the researchers have data from the same people doing different tasks, they can compare how the brain handles a quick face clip versus a complex social story in a movie.

Why Does This Matter? (The Big Picture)

The authors used this dataset to test some of the smartest computer programs (Artificial Intelligence) designed to recognize faces.

• The Surprise: They found that while these AI programs are great at recognizing faces in static photos, they fail when trying to explain how human brains process faces in moving, natural videos.
• The Takeaway: Our brains are much more sophisticated than our current computers when it comes to understanding real-life faces.

The Bottom Line

The Hyperface dataset is a gift to the scientific community. It's a "real-world" toolkit that allows researchers to stop studying faces in a vacuum and start studying them the way we actually experience them: moving, talking, and interacting. It's like finally taking the brain out of the museum and putting it back on the street corner where it belongs.

In short: They built a massive library of face videos, scanned 21 brains while people watched them, and proved that our brains are doing something much more complex than our current computers can yet understand.

Technical Summary

1. Problem Statement

Neuroimaging studies of human face processing have historically relied on controlled paradigms using small sets of static images presented under artificial conditions. While these designs allow for precise hypothesis testing, they fail to capture the richness, complexity, and ecological validity of natural face perception. Consequently, there is a significant gap in our ability to:

• Characterize face processing under naturalistic conditions.
• Test whether computational models (e.g., Deep Neural Networks) generalize to real-world scenarios.
• Benchmark AI models against human brain responses in dynamic, socially relevant contexts.

Although naturalistic neuroimaging exists, few publicly available datasets specifically target face processing with systematic variations in identity, gender, age, ethnicity, expression, and head orientation.

2. Methodology

Experimental Design & Stimuli

• Dataset Name: Hyperface.
• Participants: 21 healthy adults (11 female, mean age 27.3 ± 2.3 years).
• Stimuli: 707 unique, 4-second face video clips sampled from publicly available YouTube interviews.
  • Variability: Clips systematically vary in identity, gender, age, ethnicity, expression, and head orientation.
  • Preprocessing: Audio was removed, and distracting text was cropped.
• Behavioral Validation:
  • Rating Task: 121 Amazon Mechanical Turk (MTurk) workers rated clips on gender, age, ethnicity, emotion, number of people, and head direction. Final ratings were assigned as the modal response.
  • Arrangement Task: 39 MTurk workers performed a multi-arrangement task to generate pairwise similarity judgments based on perceived face appearance.
• Experimental Procedure:
  • Sessions: Two scanning sessions per participant.
  • Runs: Each session included 6 "Hyperface" runs (watching face clips) and 2 "Dynamic Face Localizer" runs (categorizing faces, bodies, scenes, objects).
  • Task: Participants attended to the identity of individuals. Runs were followed by an attention check (identifying if specific clips appeared in the run).
  • Counterbalancing: Stimulus order was optimized to ensure a balanced distribution of attributes across blocks.
• Integration: All 21 participants also watched The Grand Budapest Hotel (movie dataset), and 10 completed an additional controlled face identity task, allowing for within-subject cross-paradigm comparisons.

Imaging Parameters

• Scanner: 3T Siemens Magnetom Prisma with a 32-channel head coil.
• Functional Imaging (BOLD):
  • Sequence: Gradient-echo echo-planar imaging (EPI) with multiband acceleration (factor 4).
  • TR/TE: 1000/33 ms; Flip Angle: 59°.
  • Resolution: 2.5 mm³ isotropic voxels.
  • Coverage: 48 axial slices (full cortical coverage).
• Structural Imaging:
  • T1-weighted (MPRAGE): 0.9375 mm isotropic.
  • T2-weighted: 0.9375 mm isotropic.
• Diffusion Imaging: SE-EPI with 96 non-collinear directions (b=1000 s/mm²).

Data Processing & Preprocessing

• Standardization: Data converted to BIDS (Brain Imaging Data Structure) format.
• Preprocessing Pipeline: fMRIPrep 25.1.4 was used, including:
  • Intensity non-uniformity correction (N4).
  • Skull-stripping and tissue segmentation.
  • Spatial normalization to MNI152NLin2009cAsym.
  • Denoising: Regression of motion parameters, global signal, CompCor components (aCompCor/tCompCor), and polynomial trends.
  • Surfaces: Data resampled to fsaverage6 surface space.
• Quality Control: Custom Python scripts were used for denoising and calculating metrics (tSNR, ISC).

3. Key Contributions

1. Release of the Hyperface Dataset: A large-scale, naturalistic fMRI dataset featuring 707 dynamic face clips with systematic demographic and behavioral variations.
2. Comprehensive Behavioral Annotations: Includes independent observer ratings for facial attributes and pairwise similarity judgments, enabling the construction of Representational Dissimilarity Matrices (RDMs).
3. Integrated Experimental Framework: The dataset is part of a larger suite including The Grand Budapest Hotel movie data and a controlled face identity task, facilitating direct comparisons between naturalistic and controlled paradigms within the same subjects.
4. Open Science Resources:
   • Raw and preprocessed data available on OpenNeuro (ds007329, ds007384, ds007378).
   • Stimuli, behavioral data, and analysis code (including InsightFace embeddings and DNN RDMs) are publicly accessible via GitHub.

4. Results & Technical Validation

The authors validated the dataset quality across three domains:

• Motion Control: Participant motion was exceptionally low.
  • Median Framewise Displacement (FD): 0.09 mm (range 0.06–0.16 mm).
  • Motion outliers (<5% of volumes) were present in 13 of 21 participants.
• Temporal Signal-to-Noise Ratio (tSNR):
  • Mean whole-brain tSNR: 83.0 ± 4.0, comparable to other high-quality 3T datasets.
  • Spatial distribution showed expected patterns: highest in occipital/dorsal areas (near coils) and lowest in anterior temporal/orbitofrontal regions (near air-tissue boundaries).
• Inter-Subject Correlation (ISC):
  • High ISC was observed in visual areas and specific face-processing regions: Occipital Face Area (OFA), Fusiform Face Area (FFA), and Superior Temporal Sulcus (STS).
  • Significant ISC was also found in the dorsal precuneus and right inferior frontal gyrus.
  • Note: These ISC values are considered a lower bound, as they were calculated using anatomical alignment; previous work using hyperalignment on this data showed even higher correlations.

5. Significance

• Ecological Validity: Hyperface bridges the gap between artificial lab stimuli and real-world social interaction, allowing researchers to study face processing as it occurs in dynamic, natural environments.
• Computational Benchmarking: The dataset provides a rigorous testbed for evaluating Deep Neural Networks (DNNs). Preliminary findings (cited in the paper) indicate that while DNNs trained on face identification can capture behavioral categorical representations, they fail to explain human brain representations of identity during dynamic, naturalistic viewing.
• Methodological Versatility: The data supports diverse analytical approaches, including Inter-Subject Correlation (ISC), Multivoxel Pattern Analysis (MVPA), Representational Similarity Analysis (RSA), and voxelwise encoding models.
• Resource for AI and Neuroscience: By providing a unified resource across human neuroscience, computational neuroscience, and artificial intelligence, Hyperface enables the development of more accurate models of human social perception.