🧠 neuroscience

The Human Brain as a Dynamic Mixture of Expert Models in Video Understanding

This paper introduces a large-scale benchmark aligning over 100 deep video models with dynamic EEG recordings using Cross-Temporal Representational Similarity Analysis, revealing that the human brain functions as a dynamic mixture of expert models that differentially integrate temporal and static features across posterior and frontal regions.

Original authors: Sartzetaki, C., Zonneveld, A. W., Oyarzo, P., Gifford, A. T., Cichy, R. M., Mettes, P., Groen, I. I.

Published 2026-02-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sartzetaki, C., Zonneveld, A. W., Oyarzo, P., Gifford, A. T., Cichy, R. M., Mettes, P., Groen, I. I.

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

The Big Idea: The Brain is a "Smart Switchboard," Not a Single Machine

Imagine you are watching a movie. Your brain doesn't just run one single program to understand it. Instead, it acts like a dynamic switchboard that constantly flips between different tools depending on what's happening on screen.

This paper, presented at ICLR 2026, is a massive experiment where researchers tried to figure out exactly which tools the human brain uses and when it uses them. They did this by comparing the brain's electrical signals (EEG) to over 100 different computer vision models (AI systems) while people watched short, natural videos.

The New Tool: "Cross-Temporal RSA" (The Time-Traveling Detective)

Previously, scientists mostly looked at static pictures (like a single frame of a movie) to study the brain. But movies move! To study moving pictures, the researchers invented a new method called Cross-Temporal Representational Similarity Analysis (CT-RSA).

The Analogy:
Imagine you have a recording of a conversation (the brain) and a transcript of a speech (the AI model).

Old way: You tried to match the whole speech to the whole conversation at once. It was messy.
New way (CT-RSA): You act like a detective with a time machine. You take a tiny 1-second slice of the brain's activity and compare it to every possible 1-second slice of the AI's processing. You ask: "At exactly this moment, which part of the AI's brain looks most like the human brain?"

This allowed them to map the brain's activity second-by-second against the AI's internal thoughts.

The Discovery: A Four-Act Play in the Brain

The researchers found that the back of the brain (the Posterior area, near the eyes) and the front of the brain (the Frontal area, near the forehead) play very different roles.

1. The Back of the Brain: The "Shapeshifting Actor"

The back of the brain is the main stage for visual processing. It goes through four distinct acts during a 3-second video:

Act I (0.0s – 0.2s): The "Snapshot" Phase.
- What happens: The brain sees the first flash of light.
- AI Match: It matches best with static image models (like those that recognize cats in photos).
- Analogy: It's like taking a quick photo. The brain is just saying, "I see shapes and colors."
Act II (0.2s – 0.8s): The "Object Expert" Phase.
- What happens: The brain figures out what the objects are.
- AI Match: It matches best with object recognition models (like identifying a dog or a car).
- Analogy: The brain is now labeling the items in the photo. "That's a dog. That's a ball."
Act III (0.8s – 2.0s): The "Action Integrator" Phase.
- What happens: The brain starts understanding movement and context.
- AI Match: It matches best with video models that integrate time (models that understand a sequence of frames). Specifically, a new type of AI called State-Space Models (SSMs) worked best here.
- Analogy: The brain stops looking at the dog and starts watching the dog chase the ball. It needs to remember the past few seconds to understand the current moment.
Act IV (2.0s – 3.0s): The "Steady State" Phase.
- The brain maintains this understanding of the action until the video ends.

Key Insight: The back of the brain is a Mixture of Experts. It doesn't use one AI model for the whole video. It starts with a "Photo Expert," switches to an "Object Expert," and finally switches to a "Motion Expert."

2. The Front of the Brain: The "Early Bird Manager"

The front of the brain behaves differently.

What happens: It gets involved very early (around 0.2s) and then mostly stops caring about the fine details of the movement.
AI Match: It matches best with static action models (models that know what an action is, like "running," but don't necessarily track the movement frame-by-frame).
Analogy: The Frontal brain is like a manager who walks in, sees the title of the movie ("A Dog Chasing a Ball"), nods, and then goes back to paperwork. It knows the gist of the action early on but doesn't need to track every frame of the chase. It doesn't have a "time machine" connection like the back of the brain does.

What This Means for AI and the Future

The paper concludes that no single AI model is perfect at mimicking the human brain.

If you build an AI that is great at recognizing static objects, it fails at understanding video motion.
If you build an AI that is great at tracking motion, it might miss the initial details.

The Metaphor for the Future:
The human brain is like a Swiss Army Knife that changes its tool instantly.

To build better AI, we shouldn't just try to make one giant, perfect model.
Instead, we should build "Dynamic Mixture of Experts" systems. These are AI systems that can instantly switch between a "Static Vision" mode, an "Object Recognition" mode, and a "Motion Tracking" mode, just like our brains do.

Summary in One Sentence

The human brain processes video not by running one continuous program, but by rapidly switching between different "expert" modes (seeing shapes, naming objects, tracking motion), and the best way to build future AI is to copy this dynamic switching ability.

1. Problem Statement

While deep neural networks (DNNs) have successfully modeled static image processing in the human brain, significant gaps remain in understanding how the brain processes dynamic visual input (videos).

Limitations of Current Benchmarks: Previous large-scale model-brain alignment studies primarily relied on fMRI, which has poor temporal resolution (sluggish hemodynamic response). This prevents the analysis of fine-grained, millisecond-level neural dynamics essential for understanding how the brain integrates continuous visual information over time.
The Gap: There is a lack of large-scale benchmarks comparing deep video models against high-temporal-resolution brain data (EEG). Furthermore, it is unclear how different model architectures, pretraining strategies, and temporal integration capabilities align with the evolving neural representations during video perception.

2. Methodology

The authors introduce a novel framework to benchmark over 100 deep neural network models against dynamic EEG recordings of natural videos.

A. Dataset: EEG Moments Dataset (EEGMD)

Stimuli: 1,102 short (3-second) natural videos from the "Moments in Time" dataset.
Recording: High-density (128 electrodes) EEG data from 6 participants.
Resolution: Recorded at 1000 Hz, downsampled to 50 Hz for analysis.
Regions of Interest: Electrodes were partitioned into Posterior (visual cortex, 35 electrodes) and Frontal (prefrontal cortex, 54 electrodes) to investigate spatial hierarchies.

B. Model Suite

The study evaluated 110 models across four axes of variation:

Temporal Integration: Static image models (process frames independently) vs. Temporally-integrating video models (process sequences).
Classification Task: Object recognition (ImageNet) vs. Action recognition (Kinetics-400, etc.).
Architecture: CNNs, Transformers, and State-Space Models (SSMs) (e.g., VideoMamba).
Pretraining: Supervised (object/action), Self-supervised, and No pretraining.

C. Core Innovation: Cross-Temporal Representational Similarity Analysis (CT-RSA)

Standard RSA compares representations at fixed time points. The authors propose CT-RSA to handle the uncertainty of temporal alignment between model features and brain responses:

Time-Unfolding: Model features are extracted at every time step ( $t_M$ ) for every layer, creating a temporal sequence of Representational Dissimilarity Matrices (RDMs).
Cross-Temporal Matching: For every EEG time point ( $t_N$ ), the method computes the Spearman correlation ( $\rho$ ) between the brain RDM and all model RDMs across all layers and all model time steps.
Maximization: The alignment score for a specific EEG time point is the maximum correlation found across all model layers and time steps. This identifies the "best" model representation at any given moment without assuming a fixed temporal lag.
Scale: This approach generates over $10^7$ alignment scores, allowing for granular analysis of dynamic processing.

3. Key Results

The analysis reveals that the brain does not rely on a single model type but dynamically switches between different representational strategies, resembling a Mixture of Experts (MoE).

A. Temporal Stages of Processing (Posterior Electrodes)

Posterior brain activity evolves through four distinct stages during video viewing:

Stage I (0.06s – 0.24s): Static Low-Level Processing. Best aligned with early layers of static image models (e.g., AlexNet). Represents low-level visual features.
Stage II (0.24s – 0.8s): Static High-Level Object Processing. Best aligned with late layers of object recognition models (e.g., DenseNet). Represents categorical object information.
Stage III (0.8s – 2.0s): Temporally-Integrative Action Processing. A shift occurs where video models (specifically those with temporal integration) outperform static models. Best aligned with mid-level layers of action recognition models.
Stage IV (2.0s – 3.0s): Sustained action processing, though alignment scores stabilize.

Key Finding: Posterior activity shows a strong temporal correspondence: early EEG time points match early model time points, and later EEG points match later model time points (up to ~0.6s delay).

B. Frontal Electrode Dynamics

Frontal activity is early and static. Significant alignment occurs only in Stages I and II (up to 0.8s).
It is best captured by static action recognition models (late layers).
No Temporal Correspondence: Unlike posterior electrodes, there is no clear mapping between model time and EEG time in the frontal region, suggesting frontal processing extracts high-level semantic action concepts early and holds them, rather than tracking video dynamics continuously.

C. Architecture and Pretraining Effects

State-Space Models (SSMs): Models like VideoMamba show superior alignment in Stage II (Posterior), capturing mid-level representations better than CNNs or Transformers. This suggests recurrent/temporal processing mechanisms are crucial for intermediate action features.
Pretraining:
- Self-supervised pretraining yields the best alignment in Stage II (object-related).
- No pretraining yields the best alignment in Stage III (temporally-integrative action), suggesting that avoiding "shortcut learning" from unrelated pretraining tasks benefits dynamic feature extraction.

4. Key Contributions

First Large-Scale EEG Benchmark: The first study to benchmark 100+ video models against high-temporal-resolution EEG data, moving beyond the limitations of fMRI.
CT-RSA Methodology: A new analytical tool that matches time-unfolded model features to evolving brain responses, revealing dynamic prediction and temporal generalization capabilities.
Discovery of Dynamic Neural Preferences: Evidence that the brain transitions from static object processing to temporally-integrative action processing, with distinct spatial (posterior vs. frontal) and temporal characteristics.
Architectural Insights: Identification that SSMs are uniquely suited for mid-level temporal integration in the brain, and that pretraining strategies have stage-specific benefits.

5. Significance and Implications

Brain-Inspired AI: The findings suggest that a single static model cannot fully capture human video understanding. Instead, the brain operates as a Dynamic Mixture of Experts, switching between static object recognition and temporal integration strategies.
Model Design: Future AI models aiming for human-like video understanding should:
- Incorporate mechanisms to dynamically switch between semantic tasks (object vs. action) and temporal integration strategies.
- Utilize recurrent architectures (like SSMs) or feedback loops to handle intermediate temporal processing.
- Consider self-supervised pretraining for general feature extraction but potentially rely on no pretraining or specific fine-tuning for dynamic temporal tasks to avoid bias.
Theoretical Shift: Challenges the strict "temporal hierarchy" view (low-to-high features) by showing that mid-level temporally-integrative features dominate the later stages of video processing, a phenomenon invisible to fMRI due to its low temporal resolution.

In conclusion, the paper posits that the human brain's efficiency in video understanding stems from its ability to dynamically reconfigure its processing strategy, a capability that current AI models lack but could achieve through dynamic switching mechanisms inspired by this neural behavior.