💻 computer science

Learning Brain Representation with Hierarchical Visual Embeddings

This paper proposes a brain-image alignment strategy that utilizes multiple pre-trained visual encoders to capture hierarchical representations and introduces a "Fusion Prior" to enhance distributional consistency, achieving a balance between semantic retrieval accuracy and pixel-level reconstruction fidelity.

Original authors: Jiawen Zheng, Haonan Jia, Ming Li, Yuhui Zheng, Yufeng Zeng, Yang Gao, Chen Liang

Published 2026-02-10

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiawen Zheng, Haonan Jia, Ming Li, Yuhui Zheng, Yufeng Zeng, Yang Gao, Chen Liang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to play a game of "Mental Pictionary."

In this game, one person (the "Brain") looks at a picture of a golden retriever playing in a park. Instead of drawing it, they have to describe it to a "Sketch Artist" (the "AI") using only a very messy, static-filled radio signal (the "Brain Signals").

The problem is that brain signals are incredibly noisy and vague. If the Brain just says, "It’s a dog," the Artist might draw a cartoon poodle. If the Brain says, "It’s a furry thing," the Artist might draw a rug. The Artist needs more than just the "idea" of the dog; they need to know the color of its fur, the shape of its ears, and the green of the grass.

This paper, "Learning Brain Representation with Hierarchical Visual Embeddings," introduces a new way to help that Sketch Artist succeed.

The Problem: The "Semantic Gap"

Most current AI models try to decode the brain by looking only at the "Big Idea" (the semantics). They ask the brain: "What is the category of this object?" This is like trying to reconstruct a high-definition movie by only reading the plot summary. You get the story right, but you lose the beauty, the colors, and the fine details.

The Solution: The "Multi-Lens Camera"

The researchers realized that the human brain doesn't just process "ideas"; it processes a hierarchy. First, your eyes see edges and colors (the "pixels"), and then your brain assembles those into objects (the "semantics").

To mimic this, they built a Hierarchical Visual Fuser. Instead of giving the AI one single description of an image, they give it three different "lenses" to look through simultaneously:

The "Concept" Lens (CLIP): This captures the big picture—"It's a dog in a park."
The "Detail" Lens (VAE): This captures the textures and shapes—"The fur is wavy, the grass is patchy."
The "Hybrid" Lens: They fuse these together into one "Super-Description."

The Secret Sauce: The "Fusion Prior"

Even with a great description, the Sketch Artist (the AI) might still struggle to turn those words into a realistic drawing. To fix this, the researchers created a "Fusion Prior."

Think of this as a "Master Artist's Intuition." Before the AI even hears the brain signal, it has spent thousands of hours studying how colors, shapes, and objects naturally fit together. When the messy brain signal finally arrives, the AI doesn't just guess; it uses its "intuition" to bridge the gap between the noisy brain signal and a beautiful, realistic image.

Why This Matters (The Results)

The researchers tested this on real human brain data (EEG and MEG). Their results were like upgrading from a blurry charcoal sketch to a high-definition photograph:

Better Guessing: If you show the AI a brain signal and ask, "Which of these 200 images was the person looking at?" the AI is much more accurate at picking the right one.
Better Drawing: When the AI tries to reconstruct the image from scratch, the results are much clearer, with better colors and more accurate shapes.
The "Aha!" Moment: They discovered that brain signals actually contain both the big ideas and the tiny details. By feeding the AI both, they finally matched how the human visual system actually works.

In Short:

Instead of just asking the brain, "What are you thinking?", this paper teaches the AI to ask, "What are you seeing, and exactly how does it look?" This moves us one step closer to technology that can truly "see" the world through our eyes.

Technical Summary: Learning Brain Representation with Hierarchical Visual Embeddings

Published at: ICLR 2026

1. Problem Statement

The core challenge in neural visual decoding (reconstructing or retrieving images from brain signals like EEG, MEG, or fMRI) is the structural and modality gap between neural activity and visual representations.

Semantic vs. Pixel Gap: Most existing methods align brain signals primarily with high-level semantic embeddings (e.g., CLIP). While this works for category recognition, it neglects low-level pixel-level details (color, texture, edges), which are essential for high-fidelity image reconstruction.
Distributional Mismatch: Directly feeding brain-derived features into generative models (like Stable Diffusion) often leads to unstable or noisy outputs because the brain signal embeddings do not naturally follow the distribution of the visual features the generative model expects.

2. Methodology

The authors propose a two-stage framework designed to mimic the hierarchical nature of the human visual system.

A. Hierarchical Visual Fusion (HVF)
Instead of using a single encoder, the authors construct a multiscale visual representation by fusing multiple pre-trained encoders:

High-level Semantics: Uses multiple CLIP encoders (e.g., RN50, ViT-B/32) to capture abstract concepts and object relations.
Low-level Details: Incorporates a Variational Autoencoder (VAE) to capture fine-grained, pixel-level structural information.
Fusion Mechanism: A post-norm residual Hierarchical Visual Fuser (HVF) aligns these diverse embeddings into a shared $d$ -dimensional space.
Alignment: A contrastive learning objective (InfoNCE loss) is used to align the brain signal embeddings (processed via an MLP-based Brain Projection network) with this fused hierarchical visual token.

B. Pretrained Fusion Prior for Reconstruction
To enable stable image generation, the authors introduce a "Fusion Prior":

Pretraining: They train a mapping from the fused visual features to the conditioning space of a frozen diffusion model (SDXL) using large-scale visual data (ImageNet-1k). This ensures the model learns a stable, text-free way to translate visual tokens into generative guidance.
Brain-to-Fusion Alignment: Once the prior is stable, the brain encoder is trained to project brain signals into this specific, pre-trained fusion space. This prevents "representational drift" and ensures that the brain signals can effectively drive the diffusion process.

3. Key Contributions

Hierarchical Interface: A novel brain-vision interface that bridges the gap between semantic understanding and pixel-level reconstruction by fusing CLIP and VAE features.
Fusion Prior: A method to bridge the modality gap by pre-training a stable mapping on large-scale visual data, allowing for high-quality, text-free brain-to-image reconstruction.
Scientific Discovery: The study provides empirical evidence that EEG/MEG signals jointly encode both high-level semantics and low-level visual details, and that adding VAE latents is critical for performance.
Plug-and-Play Capability: The framework is modular; different brain encoders (ShallowNet, EEGNet, etc.) can be plugged into the fixed visual fusion interface.

4. Results

The method was evaluated on the THINGS-EEG and THINGS-MEG datasets across two tasks:

Retrieval (Zero-shot): The model achieved state-of-the-art (SOTA) performance. For EEG intra-subject retrieval, it reached 75.7% Top-1 accuracy, significantly outperforming the previous best (UBP at 50.9%). It also showed superior generalization in inter-subject settings.
Reconstruction: Qualitative and quantitative results (PixCorr, SSIM, and semantic similarity metrics) demonstrate that the reconstructions are more faithful to the original stimuli. The images exhibit clearer object contours and more accurate color distributions compared to baselines like ATM and CognitionCapturer.
Ablation Insights:
- Fusing RN50 + B32 + VAE yielded the best results, proving that semantics and pixels are complementary.
- Removing the VAE or B32 branches caused significant performance drops, confirming their necessity.

5. Significance

This work moves the field of neural decoding from "semantic-only" decoding toward "comprehensive visual perception" decoding. By acknowledging that the brain processes information hierarchically, the authors provide a more biologically plausible and technically robust framework. This has significant implications for both neuroscience (understanding how the brain encodes visual hierarchies) and AI (developing more effective cross-modal interfaces for brain-computer interaction).