SemVideo: Reconstructs What You Watch from Brain Activity via Hierarchical Semantic Guidance

SemVideo introduces a novel fMRI-to-video reconstruction framework that leverages a hierarchical semantic guidance module (SemMiner) to generate multi-level semantic cues, thereby achieving superior temporal coherence and visual consistency compared to existing methods.

The Gist

Imagine you are watching a movie, but instead of just seeing the screen, someone is reading your brainwaves to figure out exactly what you are seeing. This is the holy grail of "brain decoding."

For a long time, scientists could only reconstruct static pictures (like a photo of a cat) from brain activity. But trying to reconstruct a moving video (like a cat running and jumping) has been a nightmare. Previous attempts were like watching a glitchy, broken VHS tape: the cat would look like a dog in one frame, a bird in the next, and it would teleport across the screen instead of running smoothly.

Enter SemVideo, a new system that fixes these glitches by teaching the computer to "think" about the video the way a human brain does.

Here is how it works, broken down into simple analogies:

1. The Problem: The "Glitchy VHS" Effect

Imagine trying to describe a movie to a friend over a bad phone connection.

• Old Methods: You say, "It's a cat." Then, "It's a dog." Then, "It's a car." Your friend tries to draw it, but the drawing changes wildly every second. The cat's face morphs, and it jumps from left to right instantly.
• Why? The brain doesn't record every single pixel of a video. It records the gist of the story. Old computers tried to guess every pixel, which led to chaos.

2. The Solution: The "Three-Layer Storyteller" (SemMiner)

The authors realized that to fix the video, we need to give the computer a better script. They created a tool called SemMiner (Semantic Miner). Think of this as a super-smart director who breaks a movie down into three specific types of notes:

1. The Anchor (The "Who and Where"): A detailed description of the very first frame. Example: "A golden kitten sitting on a red rug." This ensures the video starts with the right character and setting.
2. The Motion Script (The "What's Happening"): A description of the action. Example: "The kitten crouches, then pounces forward, its tail flicking." This tells the computer how things move, not just what they look like.
3. The Summary (The "Big Picture"): A holistic story of the whole clip. Example: "A playful kitten exploring a living room." This keeps the overall vibe consistent.

By feeding the computer these three layers of "notes" instead of just raw pixels, the system knows exactly what to draw and how to move it.

3. The Engine: The "Brain-to-Video Translator" (SemVideo)

Once the computer has these notes, it uses a special translator to turn your brainwaves into the video. It has three main parts:

• The Semantic Decoder (The Translator): This part listens to your fMRI brain scan and says, "Ah, this part of the brain is lighting up, which means the person is thinking about a 'golden kitten'." It matches your brain activity to the "Anchor" notes.
• The Motion Adapter (The Choreographer): This is the magic sauce. It takes the "Motion Script" and tells the video generator, "Don't just draw a kitten; draw a kitten crouching and pouncing." It ensures the movement flows naturally, preventing the "teleporting" glitch.
• The Conditional Renderer (The Director): This puts it all together. It uses the "Big Picture" summary to make sure the lighting, colors, and mood stay consistent from start to finish.

4. The Result: A Clear, Smooth Movie

When they tested this on real people watching videos, the results were amazing.

• Before: The reconstructed video looked like a fever dream where objects changed shape and moved erratically.
• With SemVideo: The video clearly shows the same kitten, moving smoothly, with the right colors and actions. It's like going from a broken VHS tape to a crisp 4K streaming video.

Why This Matters

This isn't just about making cool videos. It proves that we can understand how the human brain processes complex stories and movements. By breaking the task down into "Anchors," "Motion," and "Summaries," the researchers mimicked how our own brains actually work—focusing on the key story elements rather than getting lost in the noise of every single pixel.

In short: SemVideo is like giving a blindfolded artist a detailed script, a choreographer's guide, and a director's vision, allowing them to paint a moving picture that perfectly matches what you are seeing in your mind.

Technical Summary

1. Problem Statement

Reconstructing dynamic visual experiences (videos) from brain activity (fMRI) is a significant challenge in cognitive neuroscience. While recent advances have achieved high-quality static image reconstruction, extending this to video remains difficult due to two primary shortcomings in existing methods:

1. Appearance Mismatch: Inconsistent visual representations of salient objects across frames, causing the subject of the video to change or flicker.
2. Motion Misalignment: Poor temporal coherence, resulting in jerky motion, misaligned actions, or abrupt frame transitions.

These issues arise because fMRI signals rely on the slow hemodynamic response (BOLD signal), which integrates activity over seconds, making it difficult to capture rapid motion variations. Furthermore, traditional pipelines often lack fine-grained semantic supervision, leading to "semantic under-specification" where the model fails to understand the specific actions or narrative of the video.

2. Methodology

The authors propose SemVideo, a novel framework guided by hierarchical semantic information. The core innovation is the use of a Multimodal Large Language Model (MLLM) to decompose video stimuli into multi-level textual descriptions, which then guide the decoding process.

The framework consists of three main components:

A. SemMiner: Hierarchical Semantic Guidance

Instead of treating video frames as independent images, SemMiner uses an MLLM to generate three distinct levels of semantic cues from the original video stimulus ($V$):

1. Static Anchor Description ($C_{anchor}$): Captures the visual content of the first frame (objects, colors, scene layout) to ensure basic semantic alignment.
2. Motion-Oriented Narratives ($C_{motion}$): Focuses on fine-grained dynamic cues, describing actions, transitions, and object movements.
3. Holistic Summaries ($C_{holi}$): Provides a global narrative integrating static and motion information for the entire video.

This process is a two-stage pipeline: first, a concise "core event summary" is generated to act as a semantic "rein" (preventing hallucination); second, specific prompts generate the three detailed descriptions.

B. SemVideo Decoding Framework

SemVideo transforms fMRI signals into video using three specialized modules:

1. Semantic Alignment Decoder (SAD):

   • Goal: Map fMRI signals ($X$) to the semantic feature space of the three generated descriptions ($Z(C_{anchor}), Z(C_{motion}), Z(C_{holi})$).
   • Architecture: Uses a subject-specific projector to handle varying voxel counts, followed by a subject-shared encoder (MLP + Refineformer) to align signals with CLIP-style text embeddings.
   • Training: Supervised by a combination of MSE loss, SoftCLIP contrastive loss, and a refinement loss to minimize noise and maximize semantic fidelity.

2. Motion Adaptation Decoder (MAD):

   • Goal: Reconstruct coherent motion latents from brain signals.
   • Architecture: A novel Tripartite Attention Fusion architecture that integrates:
     • Spatial Self-Attention: Captures intra-frame structure.
     • Temporal Self-Attention: Models inter-frame dependencies.
     • Semantic-Guided Cross-Attention: Explicitly injects the predicted motion semantics ($\hat{Z}(C_{motion})$) into the attention mechanism to align motion latents with semantic actions.
   • Output: Generates a sequence of motion latents ($\hat{E}(X)$) that guide the video generation.

3. Conditional Video Render (CVR):

   • Goal: Synthesize the final video using a Text-to-Video (T2V) diffusion model.
   • Strategy: A multi-stage conditional generation process:
     • The motion latents are decoded into a blurry motion sequence via a VAE.
     • The anchor feature ($\hat{Z}(C_{anchor})$) guides a Text-to-Image (T2I) model to generate a sharp initial frame.
     • The T2V model generates the final video, conditioned on the anchor frame, the motion sequence, and the holistic summary ($\hat{Z}(C_{holi})$).

3. Key Contributions

• Hierarchical Semantic Supervision: Introduced SemMiner, the first module to systematically decompose video stimuli into static, dynamic, and holistic textual descriptions to guide fMRI decoding, addressing the semantic sparsity of previous methods.
• Motion Adaptation Decoder (MAD): Proposed a tripartite attention mechanism that explicitly fuses semantic motion priors with spatial and temporal attention, significantly improving motion coherence.
• Multi-Stage Conditional Rendering: Developed a CVR strategy that progressively conditions generation on different semantic levels, ensuring both static object consistency and dynamic action smoothness.
• Neuroscience Interpretability: Validated the model using ROI-wise visualization, showing that the hierarchical components activate corresponding brain regions (e.g., motion components activating MT/MST areas), confirming the biological plausibility of the approach.

4. Experimental Results

The method was evaluated on two public datasets: CC2017 and HCP 7T.

• Quantitative Performance: SemVideo achieved State-of-the-Art (SOTA) performance on 8 out of 10 metrics, including:
  • Semantic Level: Highest 2-way/50-way retrieval accuracy and VIFI scores (semantic-video alignment).
  • Pixel Level: Competitive SSIM, PSNR, and the highest Hue-PCC (color fidelity).
  • Spatiotemporal Level: Highest CLIP similarity between adjacent frames and the lowest Endpoint Error (EPE), indicating superior motion reconstruction.
• Qualitative Results: Visual comparisons show SemVideo successfully reconstructs complex scenes (e.g., a kitten exploring, a person turning their head) with consistent object appearance and smooth motion, whereas previous methods suffer from flickering objects and jerky transitions.
• Ablation Studies: Removing any of the three semantic cues ($C_{anchor}$, $C_{motion}$, $C_{holi}$) led to significant performance drops. Specifically, removing $C_{motion}$ caused a drastic decline in temporal coherence, proving the necessity of motion-oriented semantic guidance.
• Generalization: The model demonstrated robust performance across different subjects and datasets (CC2017 and HCP), indicating strong generalization capabilities.

5. Significance

SemVideo represents a paradigm shift in brain-to-video reconstruction by moving away from pixel-level or single-frame approaches toward hierarchical semantic understanding. By mimicking how the human brain processes video (discretely focusing on keyframes and semantic narratives rather than every pixel), the framework effectively bridges the gap between slow fMRI signals and fast video dynamics.

This work not only sets a new benchmark for fMRI-to-video reconstruction but also provides a neuroscientifically grounded framework for understanding the neural mechanisms of visual perception and memory. It paves the way for future applications in brain-computer interfaces (BCI) for communication and assistive technologies for individuals with motor or speech impairments.