Bridging Brain and Semantics: A Hierarchical Framework for Semantically Enhanced fMRI-to-Video Reconstruction

Technical Summary: Bridging Brain and Semantics with CINENEURON

Problem Statement
Reconstructing dynamic visual experiences (videos) from functional magnetic resonance imaging (fMRI) signals is a critical challenge in cognitive neuroscience. While previous methods have achieved success in reconstructing static images, extending this to continuous video stimuli remains difficult due to the low signal-to-noise ratio and limited temporal resolution of fMRI data. Current state-of-the-art approaches suffer from a "semantic gap": they rely on incomplete semantic embeddings (typically limited to image-text pairs) and treat reconstruction as an isolated process using only the current stimulus. This results in videos with incorrect objects, poor action recognition, and low semantic coherence, as these methods fail to capture video-specific cues (e.g., actions, object categories) or integrate prior knowledge that the human brain naturally utilizes.

Methodology: CINENEURON
The authors propose CINENEURON, a hierarchical framework inspired by the human brain's dual-pathway processing mechanism (bottom-up sensory accumulation and top-down memory integration). The framework operates in two synergistic stages:

1. Bottom-Up Semantic Enrichment:

   • Goal: Map noisy fMRI signals into a rich embedding space that comprehensively captures textual semantics, image contents, action concepts, and object categories.
   • Mechanism: A transformer-based "Brain Model" processes fMRI signals to produce a global token and a contextual embedding. The model is trained using a suite of explicit tasks:
     • Image-Text Alignment: Aligns the global fMRI token with CLIP-derived image and text embeddings using InfoNCE loss.
     • Action Semantic Alignment: Utilizes ViCLIP to extract action embeddings from video stimuli, aligning them with the fMRI token via contrastive loss to capture temporal dynamics.
     • Category Semantic Learning: Employs Qwen2.5-VL to extract object categories from captions, which are mapped to a simplified set of superclasses. A classification head predicts these categories using a combination of Binary Cross-Entropy and Focal Loss to handle class imbalance.

2. Top-Down Memory Integration:

   • Goal: Refine the enriched fMRI embedding by dynamically selecting and fusing relevant "memories" from previously seen data, mimicking hippocampal memory replay.
   • Mechanism: A novel Mixture-of-Memories (MoM) module is introduced.
     • Retrieval: A modality-aware router dynamically assigns weights to text, image, and action embeddings stored in a memory pool (derived from training data). It computes a mixture score based on weighted similarity to the current fMRI embedding, selecting the top-k relevant memories.
     • Integration: The retrieved multimodal embeddings are fused with the fMRI embedding using cross-attention layers and zero-convolution layers. This process injects visual and action cues while preserving the stability of the generator's feature space.
   • Decoding: The final fused embedding conditions a Video Diffusion Transformer (Video DiT, specifically Wan2.1) to generate temporally consistent and semantically coherent videos.

Key Contributions

1. Hierarchical Framework: Introduction of CINENEURON, which unifies bottom-up semantic enrichment with top-down memory integration to bridge the gap between noisy brain signals and complex video content.
2. Comprehensive Semantic Enrichment: A novel approach to enriching fMRI signals by learning explicit tasks for video stimuli, capturing a broader range of semantics (image, text, action, and category) beyond traditional image-text alignment.
3. Mixture-of-Memories (MoM): A new method for dynamically selecting and fusing relevant multimodal embeddings from a memory pool to refine video decoding, addressing the isolation of current stimulus-based reconstruction.
4. State-of-the-Art Performance: Extensive experiments demonstrate that CINENEURON outperforms existing methods across multiple metrics on two challenging benchmarks.

Experimental Results
The framework was evaluated on two datasets: cc2017 (natural videos) and CineBrain (TV show episodes).

• Quantitative Performance: CINENEURON achieved superior scores in semantic-level metrics (2-way and 50-way accuracy), spatiotemporal consistency (CLIP-pcc, DTC), and motion smoothness (MS) compared to baselines like Mind-Video, NeuroClips, and CineSync. For instance, on the cc2017 dataset, it achieved a 2-way semantic accuracy of 0.850 and a CLIP-pcc of 0.972.
• Qualitative Performance: Visual comparisons show that CINENEURON faithfully reconstructs complex actions (e.g., yoga, petting) and object categories, whereas baselines often hallucinate incorrect objects or fail to capture specific movements.
• Human Evaluation: In a study with 20 participants, CINENEURON was preferred over competing methods in 63.77% of cases for semantic alignment and 67.59% for overall fidelity.
• Ablation Studies: Results confirm that each component (action alignment, category learning, and MoM) contributes significantly to performance. Notably, the inclusion of hippocampus fMRI data further improved semantic metrics, validating the top-down memory integration design.

Significance and Claims
The paper claims that CINENEURON represents a significant advancement in brain decoding by moving beyond isolated stimulus-response mapping to a system that mimics the brain's use of prior knowledge and memory. By explicitly integrating action and category semantics and utilizing a memory retrieval mechanism, the method produces videos that are not only visually coherent but also semantically accurate. The authors posit that this approach offers deeper insights into neural processing, demonstrating functional alignment between the model's learned representations and known visual cortical areas (e.g., lateral occipital and ventral cortices). The work suggests that future progress in fMRI-to-video reconstruction relies on such hierarchical, semantically enriched frameworks rather than simple alignment of raw signals.