Multisensory coding of audiovisual movies in the human hippocampus

Using high-resolution fMRI, this study reveals that while univariate hippocampal activation is primarily visual, multivariate analyses demonstrate robust auditory and multisensory coding in the human hippocampus, characterized by multisensory facilitation in the posterior region and crossmodal generalization in the anterior region.

The Gist

Imagine your brain's hippocampus as the brain's ultimate "librarian." For decades, scientists thought this librarian only cared about books (visual information). They believed if you showed the librarian a picture of a cat, they'd organize it, but if you played a recording of a cat meowing, the librarian would just shrug and ignore it.

This new study asks: What if the librarian actually listens to the audio too? And what happens when the picture and the sound match up?

Here is the story of what the researchers found, broken down into simple concepts.

1. The Experiment: The "Movie Mixer"

The researchers didn't just show people pictures or play sounds; they treated the brain like a movie theater. They showed 30 people short clips of nature scenes (like waves crashing or wolves howling) in four different ways:

• The Silent Movie: Just the video, no sound.
• The Radio Play: Just the sound, no video.
• The Perfect Match: The video and sound were synced correctly (a real movie).
• The Glitch: The video of a wolf was paired with the sound of a race car (a mismatch).

They watched the participants' brains using a super-powerful MRI scanner that could see tiny details deep inside the hippocampus.

2. The Big Surprise: The "Invisible" Sound

When the researchers looked at the average volume of brain activity (like checking if the lights in the library turned on), they found something boring:

• The lights turned on for the Silent Movies (Visual).
• The lights stayed off for the Radio Plays (Auditory).

It looked like the hippocampus was ignoring the sound completely. This is what most scientists expected.

But then, they looked closer.

Instead of just checking the "volume," they looked at the pattern of activity (like checking the specific arrangement of books on the shelves). When they did this, they found a secret code:

• Even though the "lights" didn't turn on for the sound, the arrangement of the books was different for every single movie clip.
• Translation: The hippocampus was listening to the sounds and organizing them, but it was doing it so subtly that the old "volume check" method missed it. It's like a librarian who silently reorganizes the audio section without turning on the main light switch.

3. The Two Zones of the Librarian

The hippocampus isn't just one room; it has a back section (Posterior) and a front section (Anterior). The study found they do very different jobs:

The Back Section: The "Detail Detective"

• What it does: When the video and sound matched perfectly (the "Perfect Match" movie), the back section got really excited.
• The Analogy: Imagine a detective who solves a case faster when they have both a fingerprint (visual) and a voice print (audio). The back section of the hippocampus loves it when the senses agree. It helps you remember the scene more clearly because the sound and sight are reinforcing each other. This is called Multisensory Facilitation.

The Front Section: The "Abstract Philosopher"

• What it does: The front section didn't care if the sound and video matched. Instead, it realized that the sound of a wolf and the sight of a wolf are the same thing.
• The Analogy: Imagine a philosopher who understands that "Wolf" is a concept. Whether you see a wolf or hear a wolf, the front section says, "Ah, that's a wolf." It strips away the details (is it a big wolf? is it a loud wolf?) and focuses on the essence of the object. This is called Crossmodal Transfer. It allows your brain to generalize: "I know this sound belongs to this visual scene."

4. The "Glitch" Effect

When they played the mismatched movies (Wolf video + Race car sound), the brain didn't get confused in a chaotic way.

• In the Back Section, the "Perfect Match" was still the best. The mismatch didn't help.
• In the Front Section, the brain still recognized the abstract concept, but the "glitch" didn't break the system.
• Interestingly, other parts of the brain (the "sensory cortex") got very confused by the mismatch, showing that the hippocampus is actually quite good at keeping things organized even when the world feels weird.

The Takeaway

This study changes how we see the hippocampus:

1. It's not just for eyes: It processes sound just as well as sight, but we needed better tools to see it.
2. It has a split personality:
   • The Back helps you remember specific, detailed moments when your senses agree (great for memory).
   • The Front helps you understand the general idea of things, regardless of how you experience them (great for learning and generalizing).

In short: The hippocampus is the brain's master organizer. It doesn't just store visual movies; it listens to the soundtrack, checks if the audio and video match to boost your memory, and figures out the "big picture" meaning of what you are experiencing.

Technical Summary

1. Problem Statement

While the human hippocampus receives convergent input from all sensory systems and is known to encode multisensory associations in non-human animals, human research has historically focused almost exclusively on the visual modality. This creates a fundamental gap in understanding:

• How the human hippocampus represents non-visual (specifically auditory) information.
• How it integrates information across modalities (multisensory integration).
• Whether the hippocampus exhibits multisensory facilitation (enhanced processing when stimuli are congruent) and crossmodal generalization (abstract representations invariant to modality).

2. Methodology

Experimental Design & Stimuli

• Participants: 30 healthy young adults (N=30) underwent high-resolution fMRI.
• Stimuli: 10 short (6-second) naturalistic movie clips.
• Conditions: Each clip was presented in four formats:
  1. Visual Only: Video track isolated.
  2. Auditory Only: Audio track isolated.
  3. Congruent Audiovisual: Original synchronized audio and video.
  4. Incongruent Audiovisual: Audio from one movie paired with video from another (mismatched).
• Procedure: Participants viewed multiple repetitions of these clips across 9 runs. A cover task (button press at movie offset) ensured attention. A pre-exposure phase familiarized participants with the clips to minimize novelty-driven memory demands, isolating representational coding.

Imaging & Preprocessing

• Scanner: Siemens Prisma 3T with a 64-channel head coil.
• Sequence: High-resolution EPI (1.5 mm isotropic voxels, TR=1500ms, multiband factor=6).
• Audio: Active noise cancellation (OptoActive headphones) and passive earplugs were used to ensure high-fidelity auditory delivery.
• Preprocessing: Standard pipeline (fMRIPrep) including motion correction, susceptibility distortion correction, and normalization to MNI space.

Analytical Approach
The study employed a dual-analysis strategy to distinguish between mean activation and distributed pattern coding:

1. Univariate Analysis (GLM): General Linear Models estimated mean beta activation for each condition across hippocampal Regions of Interest (ROIs).
   • ROIs: Whole hippocampus, Anterior (aHPC) vs. Posterior (pHPC) subdivisions, and subfields (CA1, CA2/3, DG, Subiculum).
2. Multivariate Analysis (RSA): Representational Similarity Analysis using a leave-one-run-out cross-validation approach.
   • Movie-Specific Pattern Similarity (MSPS): Calculated as the difference between within-movie similarity (diagonal) and between-movie similarity (off-diagonal) to quantify how distinctively each movie was represented.
   • Crossmodal Transfer: Correlation of patterns between Auditory Only and Visual Only presentations of the same movie to test for modality-invariant (abstract) representations.
   • Whole-Brain Searchlight: Applied RSA across the entire brain to compare hippocampal effects with cortical multisensory integration sites (e.g., STS, pSTG).

3. Key Results

Univariate Activation (Mean Signal)

• Visual: Robust activation across all hippocampal subfields and subdivisions for visual stimuli.
• Auditory: No significant univariate activation was detected for auditory-only stimuli in any hippocampal region.
• Multisensory: No evidence of multisensory facilitation (Congruent > Visual) in univariate activation.
• Conclusion: Standard mean activation analysis suggests the hippocampus is visually dominant and insensitive to auditory input.

Multivariate Representations (Pattern Similarity)

• Auditory Coding: Despite the lack of univariate activation, RSA revealed robust, movie-specific representations for auditory-only stimuli in the whole hippocampus, anterior hippocampus, and subiculum. This indicates auditory information is encoded in distributed voxel patterns.
• Multisensory Facilitation (Posterior Hippocampus):
  • The posterior hippocampus showed significantly enhanced MSPS for congruent audiovisual stimuli compared to both unisensory conditions.
  • This suggests the posterior hippocampus facilitates visual processing when a matching auditory track is present.
• Crossmodal Generalization (Anterior Hippocampus):
  • The anterior hippocampus exhibited significant crossmodal transfer: patterns for the same movie in auditory and visual modalities were more similar to each other than to patterns from different movies.
  • The posterior hippocampus did not show this effect.
  • This suggests the anterior hippocampus maintains abstract, modality-invariant representations.
• Whole-Brain Correlates: Searchlight analyses confirmed parallel effects in cortical regions known for multisensory integration (Superior Temporal Sulcus, posterior STG), validating the paradigm.

Control Analyses

• Effects did not strengthen with stimulus repetition, ruling out memory retrieval or mental imagery as the primary driver of the observed patterns.
• Incongruent stimuli did not show enhanced pattern similarity in the hippocampus, confirming sensitivity to congruence.

4. Key Contributions

1. Discovery of Auditory Coding: Provided the first high-resolution fMRI evidence that the human hippocampus encodes complex auditory scenes, a finding invisible to univariate analysis but detectable via multivariate pattern analysis.
2. Functional Dissociation along the Long Axis: Identified a clear functional gradient:
   • Posterior Hippocampus: Specialized for fine-grained, multisensory facilitation (enhancing visual processing via congruent audio).
   • Anterior Hippocampus: Specialized for abstract, crossmodal generalization (integrating audio and video into a unified, modality-invariant concept).
3. Methodological Insight: Demonstrated that relying solely on univariate activation in the hippocampus leads to false negatives regarding non-visual processing, emphasizing the necessity of multivariate approaches (RSA) for studying sensory integration.

5. Significance

This study fundamentally advances the understanding of the human hippocampus by moving beyond its traditional role as a visual memory structure. It establishes that the hippocampus is a domain-general relational coding system capable of:

• Representing complex auditory scenes.
• Binding sensory inputs when they are congruent (facilitation).
• Abstracting information across sensory modalities (generalization).

The findings bridge the gap between animal models (which show multisensory hippocampal neurons) and human neuroimaging, suggesting that the hippocampus plays a critical role in constructing coherent, multisensory experiences and memories from fragmented sensory inputs. The identified posterior-to-anterior gradient offers a new framework for understanding how the hippocampus transitions from processing specific sensory details to forming abstract, generalized representations.