How does longer temporal context enhance multimodal narrative video processing in the brain?

Technical Summary: Enhancing Multimodal Narrative Video Processing via Longer Temporal Context

Problem Statement
Understanding how humans and artificial intelligence systems process complex narrative videos is a fundamental challenge at the intersection of neuroscience and machine learning. While prior work has demonstrated that contextual representations from Transformer-based models improve brain alignment for text and speech, the role of longer temporal context in brain alignment for naturalistic multimodal stimuli (video and audio) remains underexplored. Existing multimodal brain-encoding studies typically rely on short clips (e.g., 1.49s to 4s), often neglecting the extended temporal integration required for narrative comprehension. Humans seamlessly integrate visual scenes and story context over time, a process that recruits a hierarchy of cortical regions operating on multiple timescales. Current AI models, constrained by compute and memory, often process videos as limited frame sequences, potentially missing the long-timescale dynamics necessary to model cortical responses during narrative understanding.

Methodology
The authors investigate how temporal context length (3s to 24s) and narrative-task prompting shape brain-model alignment using fMRI data from participants watching full-length movies.

• Dataset: The study utilizes the Movie10 dataset (Boyle et al., 2020), a multimodal naturalistic fMRI dataset containing recordings from four subjects watching four movies (The Bourne Supremacy, The Wolf of Wall Street, Hidden Figures, and Life). The "Life" movie repetitions were used for testing, while the other movies served as training data.
• Models:
  • Multimodal Large Language Models (MLLMs): Two pretrained video-audio models were evaluated: Qwen-2.5-Omni and DATE (Dynamic Absolute Time Enhancement). These are end-to-end models that process video and audio in an interleaved manner.
  • Unimodal Baselines: Two unimodal video models, TimeSFormer and VideoMAE, were used for comparison.
• Experimental Design:
  • Temporal Context: Video clips were extracted using a sliding window approach with durations of 3s, 6s, 9s, 12s, 18s, and 24s.
  • Narrative Tasks: Four specific task prompts were applied to the MLLMs: Character Motivation, Event Boundary Detection, Multi-Scene Summary, and Narrative Summary.
  • Encoding Models: Bootstrap ridge regression models were trained to predict voxel-wise fMRI activity from the model-derived features. Performance was measured using Pearson Correlation, normalized by cross-subject prediction accuracy to account for biological noise.
  • Ablation Studies: The study included ablations for modality (video-only vs. video+audio), prompting (with vs. without instructions), and frame sampling density (constant 1 fps vs. scaled sampling) to isolate the drivers of performance gains.

Key Contributions and Results

1. Temporal Context Benefits MLLMs but not Unimodal Models:
   Increasing clip duration from 3s to 24s substantially improved brain predictivity for video-audio MLLMs (Qwen-2.5-Omni showed a ~41% relative gain; DATE showed ~33.4%). In contrast, unimodal video baselines (TimeSFormer, VideoMAE) showed little to no improvement with longer context windows. This suggests that the gains arise from the integration of temporal and multimodal information rather than simply adding more frames.

2. ROI-Specific Temporal Gradient:
   The study identified a hierarchical alignment pattern mirroring cortical processing:

   • Shorter windows (3–6s): Optimally aligned with perceptual and early language regions (e.g., Posterior Temporal Lobe - PTL).
   • Longer windows (12–24s): Preferentially aligned with higher-order integrative regions (e.g., Posterior Cingulate Cortex - PCC, dorsomedial Prefrontal Cortex - dmPFC, Anterior Temporal Lobe - ATL).
   • This gradient was consistent across both MLLMs and was not observed in unimodal baselines.

3. Layer-to-Cortex Hierarchy:
   In MLLMs, deeper layers (29–36) showed better alignment with higher-order cortical regions, while early layers (1–12) aligned with early sensory regions. This layer-wise hierarchy in the models paralleled the temporal receptive window hierarchy observed in the human cortex.

4. Task-Dependent Representations:
   Narrative-task prompts elicited distinct, region-specific alignment patterns:

   • Narrative and Multi-Scene Summaries: Dominated alignment in higher-order language and association ROIs (e.g., IFG, MFG, AG, PCC), reflecting global semantic integration.
   • Character Motivation: Preferentially aligned with lateral/ventral temporal and occipito-temporal regions, suggesting agent-centric inference grounded in audiovisual content.
   • Event Boundary: Showed sparse, localized alignment, primarily in dmPFC.
   • The study confirmed that these task-specific patterns are robust to paraphrasing and reflect genuine task-conditioned semantic structures.

5. Stability vs. Shift in Voxel Tuning:
   Analysis of the top-ranked video clips driving voxel responses revealed that:

   • Visual ROIs (e.g., LOC, face-selective cortex) maintained stable clip preferences across different temporal windows and tasks.
   • Higher-order ROIs (e.g., AG, PCC) exhibited significant shifts in the clips that maximally activated them as temporal context and task instructions changed, indicating sensitivity to broader semantic and narrative context.

Significance and Claims
The paper positions long-form narrative movies as a principled testbed for studying long-timescale temporal integration in long-context MLLMs and its relationship to cortical responses. The authors claim that their work provides one of the first systematic evaluations of extended temporal context and narrative-task prompting in video-audio MLLMs.

Key conclusions include:

• Context and Task as Probes: Temporal context and narrative-task prompting act as complementary probes for understanding long-form narrative video processing in the brain and for interpreting long-context representations in MLLMs.
• Representational Hierarchy: Video-audio MLLMs capture a representational hierarchy and temporal integration that tracks cortical processing during narrative comprehension, a capability that unimodal models lack.
• Functional Specialization: Task conditioning serves as a functional probe, revealing that different instructions yield distinct, ROI-specific patterns of brain alignment rather than a single generic embedding.

The study acknowledges limitations, including the maximum temporal context tested (up to 24s, though naturalistic comprehension may require tens of seconds to minutes) and the scope of the dataset (four subjects, English-language films). However, the findings establish a clear link between the temporal integration capabilities of modern MLLMs and the hierarchical temporal receptive windows of the human brain.