💻 computer science

ViBE: Visual-to-M/EEG Brain Encoding via Spatio-Temporal VAE and Distribution-Aligned Projection

The paper proposes ViBE, a novel brain encoding framework that utilizes a spatio-temporal convolutional variational autoencoder (TSC-VAE) and a Q-Former to map visual features into a distribution-aligned latent space, enabling the effective generation of high-quality MEG and EEG signals from visual stimuli.

Original authors: Ganxi Xu, Zhao-Rong Lai, Yuting Tang, Yonghao Song, Shuyan Zhou, Guoxu Zhou, Boyu Wang, Jian Zhu, Jinyi Long

Published 2026-04-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ganxi Xu, Zhao-Rong Lai, Yuting Tang, Yonghao Song, Shuyan Zhou, Guoxu Zhou, Boyu Wang, Jian Zhu, Jinyi Long

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Reading Minds from Pictures

Imagine you have a camera that can take a picture of what someone is looking at, and you want to know exactly what their brain is "seeing" at that moment. This is the goal of Brain Encoding.

Currently, scientists can do this with fMRI (brain scans), but those machines are slow, expensive, and bulky. The authors of this paper want to do it with EEG and MEG—sensors that sit on the scalp like a headset. These are fast and portable, but the signals they pick up are messy and complex.

The paper introduces ViBE, a new AI system that takes a picture (visual stimulus) and predicts exactly what the brain's electrical signals (M/EEG) would look like in response to that picture.

The Problem: The "Translation" Gap

The authors point out that previous attempts to do this had two main problems:

The "One-Size-Fits-All" Mistake: Old models tried to guess the brain's reaction with a single, fixed answer. But brains are messy! Two people (or even the same person twice) might react slightly differently to the same picture. Old models averaged these differences out, losing the nuance.
The "Language Barrier": Imagine trying to translate a poem written in English (the picture) into a song written in a completely different language with a different musical scale (the brain signal).
- The Picture side: Uses "CLIP," a smart AI that understands images. Its "language" is very compact and precise.
- The Brain side: Uses a model to understand brain waves. Its "language" is huge, messy, and spreads out over time and space.
- The Issue: Previous methods tried to force these two languages to match directly, which didn't work well because they were on completely different scales.

The Solution: ViBE (Visual-to-M/EEG Brain Encoding)

The authors built a two-stage factory to solve this. Think of it as a Translation and Reconstruction Pipeline.

Stage 1: The "Brain Signal Architect" (TSC-VAE)

Before we can translate a picture into a brain signal, we need to understand what a brain signal actually looks like.

The Analogy: Imagine the brain signal is a complex 3D sculpture made of clay. Previous tools tried to flatten this sculpture into a 2D drawing, losing all the depth.
What ViBE does: They built a special tool called TSC-VAE. Instead of flattening the sculpture, this tool learns the hierarchical structure of the clay.
- It understands that brain signals have a time dimension (how the signal changes second by second) and a space dimension (how different parts of the brain light up).
- Crucially, they realized the brain processes information in layers (like peeling an onion). So, their tool peels the layers gently rather than smashing them all at once.
The Result: This tool creates a perfect "blueprint" (a latent space) of what a real brain signal looks like. It's so good at it that it can reconstruct the original brain signal with high accuracy.

Stage 2: The "Universal Translator" (Q-Former)

Now that we have a perfect blueprint of the brain signal, we need to turn a picture into that blueprint.

The Analogy: You have a photo (the input) and a blueprint (the target). But the photo is tiny and neat, while the blueprint is massive and sprawling. If you just stretch the photo, it looks distorted.
What ViBE does: They use a component called Q-Former. Think of this as a master translator who knows both languages.
- It takes the "English poem" (the picture features from CLIP).
- It expands and reshapes it to match the "musical scale" of the brain blueprint.
- It creates a "Neural Proxy"—a fake brain signal that looks like the real thing but is generated from the picture.

The Secret Sauce: Two Types of Alignment

To make sure the translation is perfect, they use two different rules to check the work:

Point-by-Point Check (MSE): "Does this specific pixel in the fake signal match the specific pixel in the real signal?"
Vibe Check (Sliced Wasserstein Distance): This is the clever part. Even if the pixels don't match exactly, does the overall shape and distribution of the fake signal feel the same as the real one? It's like checking if a fake painting has the same feeling and color balance as the original, even if the brushstrokes aren't identical.

The Results: Does it Work?

The team tested ViBE on two massive datasets (THINGS-EEG2 and THINGS-MEG) where people looked at thousands of images while wearing brain sensors.

The Score: ViBE significantly outperformed all previous methods.
The Analogy: If previous methods were like a student guessing the answer on a math test and getting it 40% right, ViBE got it over 60% right (in terms of correlation).
The "Scale" Discovery: They found that the gap between the picture language and the brain language is huge (about 40 times different in size). Their "Translator" (Q-Former) managed to bridge most of that gap, making the translation much more accurate.

Summary

ViBE is a new system that acts like a high-tech translator. It first learns the complex, layered structure of brain signals (Stage 1), and then uses a smart translator to convert images into those specific brain signal patterns (Stage 2). By checking both the exact details and the overall "vibe" of the signals, it creates a much more accurate prediction of what the brain is thinking when it sees an image.

This is a step forward for visual prosthetics (devices that might one day restore sight to the blind), as it shows we can generate high-quality brain signals from images, which is the first step in teaching a device how to "speak" to a brain.

1. Problem Definition

The paper addresses the challenge of Brain Encoding, specifically the task of predicting neural responses (Magnetoencephalography - MEG, and Electroencephalography - EEG) from visual stimuli (images).

Context: This technology is crucial for understanding human visual perception and developing visual prostheses for patients with incurable blindness.
Limitations of Existing Methods:
- Deterministic Models: Previous approaches often use deterministic generative strategies that map visual features to a single fixed latent representation. This fails to capture the inherent "one-to-many" nature of brain responses, leading to averaged, physiologically invalid outputs.
- Modality Gap: There is a significant feature scale discrepancy between visual embeddings (e.g., from CLIP) and neural embeddings (e.g., fMRI or M/EEG). Direct alignment often fails because the scales differ by orders of magnitude.
- Distribution Mismatch: Existing methods often rely solely on Mean Squared Error (MSE) for point-wise alignment, neglecting the alignment of probability distributions between visual semantics and neural responses.
- Data Constraints: Most prior work focuses on fMRI, which has low temporal resolution and high cost, making it less suitable for Brain-Computer Interfaces (BCIs) compared to M/EEG.

2. Methodology: The ViBE Framework

The authors propose ViBE, a two-stage framework designed to generate high-fidelity M/EEG signals from images.

Stage I: Neural Response Reconstruction (TSC-VAE)

The first stage focuses on learning a high-fidelity latent space for M/EEG signals.

Architecture: A Spatio-Temporal Convolutional Variational Autoencoder (TSC-VAE).
Innovation (TSConvPlus): Unlike standard temporal-spatial convolutions (TSConv) that collapse the entire spatial dimension in one step, ViBE introduces TSConvPlus. It uses a spatial kernel size of $(k_s, 1)$ $(k_{s}, 1)$ where $k_s < C$ $k_{s} < C$ (number of channels).
- Rationale: This respects the hierarchical nature of visual processing (V1 $\to$ V2 $\to$ V4 $\to$ IT cortex), allowing the model to capture hierarchical spatial information across different brain regions rather than collapsing them immediately.
Training: The encoder maps M/EEG signals to a latent distribution, and the decoder reconstructs the signal. The model is optimized using the Evidence Lower Bound (ELBO), combining reconstruction loss (MSE) and KL divergence regularization.

Stage II: Visual-to-Neural Mapping (Q-Former Projection)

The second stage bridges the gap between visual features and the learned neural latent space.

Feature Extraction: A frozen CLIP visual encoder extracts image embeddings.
Bridging the Gap: A Q-Former module is employed to map CLIP image embeddings to the same dimensional space as the TSC-VAE latent embeddings.
- It uses learnable query tokens that interact with CLIP embeddings via cross-attention.
- A projection head reshapes the output into Neural Proxy Embeddings ( $\hat{z}$ ).
Alignment Strategy: To ensure comprehensive alignment, the authors combine two loss functions:
1. Mean Squared Error (MSE): For point-wise feature matching.
2. Sliced Wasserstein Distance (SWD): For probability distribution alignment. This addresses the limitation of MSE by ensuring the distribution of the generated neural proxies matches the distribution of the ground truth latent embeddings.
Optimization: The Q-Former and projection head are trained to minimize the combined loss ( $\mathcal{L}_{Align} = \mathcal{L}_{MSE} + \lambda\mathcal{L}_{SWD}$ ) while the TSC-VAE and CLIP encoders remain frozen.

Inference Pipeline

Given a test image:

Extract CLIP image embeddings.
Project through the trained Q-Former to generate Neural Proxy Embeddings.
Decode via the frozen TSC-VAE decoder to generate the predicted M/EEG signal.

3. Key Contributions

TSC-VAE Architecture: Proposed a novel Variational Autoencoder with TSConvPlus that effectively captures the hierarchical spatio-temporal characteristics of M/EEG signals, enabling high-fidelity reconstruction.
Modality Gap Mitigation: Introduced the use of Q-Former to map CLIP embeddings to the neural latent space, effectively bridging the massive feature scale discrepancy (approx. 2 orders of magnitude) between visual and neural modalities.
Dual-Alignment Loss: Demonstrated that combining MSE (point-wise) with Sliced Wasserstein Distance (distributional) yields superior cross-modal alignment compared to using MSE alone.
Empirical Validation: Validated the framework on the THINGS-EEG2 and THINGS-MEG datasets, achieving state-of-the-art performance in generating brain signals from images.

4. Experimental Results

The model was evaluated on THINGS-EEG2 (10 subjects) and THINGS-MEG (4 subjects) using MSE, Pearson Correlation, and Cosine Similarity.

Performance: ViBE significantly outperformed existing baselines (including SynBrain, MindSimulator, and previous diffusion-based approaches).
- THINGS-EEG2 (Within-subject): Achieved a Pearson correlation of 0.635 (vs. 0.425 for the previous best, Xu et al.).
- THINGS-MEG (Within-subject): Achieved a Pearson correlation of 0.543 (vs. 0.379 for the previous best).
Ablation Studies:
- TSConvPlus vs. TSConv: Using TSConvPlus improved Stage I reconstruction Pearson correlation from ~0.29 to 0.94 (EEG) and ~0.62 to 0.98 (MEG), proving the importance of hierarchical spatial processing.
- Loss Function: The combination of MSE + SWD yielded the best Pearson and Cosine scores, confirming that distributional alignment is critical even if it slightly increases MSE due to amplitude scaling differences.
- Brain Regions: Ablation studies confirmed that the Occipital region has the strongest influence on performance, and a "Visual Pathway" condition (retaining only Temporal and Occipital channels) performed slightly better than the full model, supporting the ventral stream hypothesis.
Cross-Subject Generalization: While cross-subject performance dropped (as expected), the Leave-One-Subject-Out protocol showed that training on multi-subject data significantly improves generalization compared to training on single-subject data.

5. Significance

Advancement in Brain-Computer Interfaces (BCIs): By focusing on M/EEG (high temporal resolution) rather than fMRI, ViBE provides a more practical foundation for real-time visual prostheses.
Biological Plausibility: The framework generates neural responses that are not just statistically correlated but distributionally aligned with ground truth, offering a more realistic simulation of brain activity for patients with blindness.
Methodological Insight: The paper highlights the critical importance of handling feature scale discrepancies and distribution alignment in cross-modal brain encoding tasks, offering a blueprint for future research in visual-to-neural synthesis.