💻 computer science

Hi-DREAM: Brain-Inspired Hierarchical Diffusion for fMRI-to-Image Reconstruction via ROI Encoder and VisuAl Mapping

Hi-DREAM is a brain-inspired hierarchical diffusion framework that reconstructs natural images from fMRI data by leveraging a multi-scale cortical pyramid of Region of Interest (ROI) streams to inject anatomy-aware priors into a U-Net, achieving state-of-the-art performance with improved semantic fidelity and interpretable contributions from different visual areas.

Original authors: Guowei Zhang, Yun Zhao, Kai Sun, Moein Khajehnejad, Adeel Razi, Dinh Phung, Levin Kuhlmann

Published 2026-07-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Guowei Zhang, Yun Zhao, Kai Sun, Moein Khajehnejad, Adeel Razi, Dinh Phung, Levin Kuhlmann

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine your brain is a massive, bustling construction site. When you look at a picture of a cat, different teams of workers (brain regions) start building the image in your mind. Some workers are focused on the basic blueprints and edges (early vision), others are assembling the body parts and fur texture (middle vision), and a final team is deciding, "Yes, this is definitely a cat, not a dog" (late vision).

For a long time, scientists trying to turn brain scans (fMRI) back into pictures have been like a boss who shouts a single, vague instruction to the whole construction site: "Build something!" This often results in a messy building that might have the right idea of a cat, but the wrong shape or color.

Hi-DREAM is a new, smarter way to run this construction site. Here is how it works, broken down simply:

1. The Problem: One Size Doesn't Fit All

Previous methods took all the brain activity, mashed it into one single "summary" number, and fed it to an AI image generator. It's like giving a chef a single word ("Dinner!") and expecting them to cook a perfect meal without knowing if you want soup or steak, or if you want it spicy or sweet. The AI had to guess everything, often missing the fine details or the specific meaning of what you saw.

2. The Solution: The Specialized Assembly Line

The researchers, Guowei Zhang and his team, realized that the brain works in layers. They built a system called Hi-DREAM that respects this natural hierarchy. Instead of one big shout, they send three distinct, specialized instructions to the AI, matching the brain's own workflow:

The "Blueprint" Team (Early ROIs): They look at the brain areas that see edges and shapes. They tell the AI, "Make sure the outline is sharp and the layout is correct."
The "Parts" Team (Middle ROIs): They look at areas that see contours and object parts. They tell the AI, "Now, add the specific shapes, like the curve of a nose or the petals of a flower."
The "Meaning" Team (Late ROIs): They look at areas that understand categories. They tell the AI, "Make sure this looks like a cat, not a dog, and get the colors right."

3. The Magic Tool: The "Smart Adapter"

How do they get these instructions to the AI? They use a clever tool called a ROI Adapter.
Think of this adapter as a translator who takes the brain's raw signals and organizes them into a multi-scale pyramid.

The "Blueprint" instructions go to the AI's shallow layers (where it draws the basic structure).
The "Parts" instructions go to the middle layers.
The "Meaning" instructions go to the deep layers (where it refines the details and identity).

This ensures that the right type of information arrives at the right stage of the drawing process. It's like a conductor ensuring the violin section plays the melody while the drums keep the beat, rather than having everyone play the same note at the same time.

4. The Result: A Clearer, Smarter Picture

When they tested this on the Natural Scenes Dataset (NSD)—a huge collection of brain scans while people looked at natural photos—the results were impressive:

Better Meaning: The images were much better at capturing the identity of the object (e.g., correctly identifying a specific type of flower or a piano).
Better Structure: The images kept the correct shapes and layouts, avoiding the "melting" or "drifting" colors seen in older methods.
Interpretability: Because the system is organized by brain regions, the researchers can actually see which part of the brain contributed to which part of the image. If the image looks weird, they can check if the "Meaning" team or the "Blueprint" team made a mistake.

What It Doesn't Do (Yet)

The paper is honest about its limits. While Hi-DREAM is great at getting the big picture and the main objects right, it still struggles with tiny, fine details like the texture of fur or the exact shape of a face if the brain signal is weak. It also currently works best for the specific person whose brain was scanned, rather than being a universal "one-size-fits-all" decoder for everyone.

In a nutshell: Hi-DREAM stops treating the brain like a black box that spits out one random idea. Instead, it listens to the brain's different departments, organizes their specific instructions, and hands them to an AI at the exact moment they are needed, resulting in a much clearer and more accurate reconstruction of what the person saw.

Technical Summary: Hi-DREAM

Problem Statement

Reconstructing natural images from functional Magnetic Resonance Imaging (fMRI) data requires bridging neural activity with the structural and semantic representations utilized by modern generative models. While recent diffusion-based decoders have improved visual quality, a critical limitation persists: most existing methods compress fMRI signals into a single global embedding before conditioning the generative model. This approach largely ignores the hierarchical organization of the human visual cortex, making it difficult to inspect how different cortical regions contribute to specific aspects of the reconstruction (e.g., local structure vs. high-level semantics). Furthermore, treating the visual cortex as a monolithic entity fails to leverage the distinct roles of early (retinotopic), middle (contour/parts), and late (semantic) visual areas at different stages of the image generation process.

Methodology

The authors propose Hi-DREAM (Brain-Inspired Hierarchical Diffusion), a framework that structures fMRI conditioning according to the visual hierarchy. The method consists of two primary components integrated into a latent diffusion backbone:

1. Hierarchical ROI Adapter

Instead of a single embedding, Hi-DREAM groups fMRI responses into three distinct streams based on visual Regions of Interest (ROIs):

Early Stream: Includes V1/V2, associated with local and retinotopic structural information.
Middle Stream: Includes V3/hV4, associated with contours and object parts.
Late Stream: Includes LOC/FFA, associated with category-level semantics.

The ROI Adapter converts these trial-wise responses and subject-specific masks into a multi-scale cortical pyramid. It employs:

Scale-space construction: Soft ROI masks are downsampled to match U-Net resolutions (64, 32, 16).
Learnable gating: Depth-dependent weights ( $w_{r,s}$ ) and mixing weights ( $\alpha_{g,s}$ ) allow each U-Net scale to receive a specific mixture of early, middle, and late evidence.
Multi-Head Latent Attention (MHLA): This module models cooperative interactions among the three streams, allowing information from one ROI group to influence how others are utilized at specific denoising depths.

2. ROI-Conditioned ControlNet

A lightweight ControlNet branch injects the anatomy-aware priors generated by the adapter into the diffusion backbone.

Depth-Matched Injection: The ControlNet translates the cortical pyramid into hints ( $\xi_s$ ) that are injected into corresponding U-Net depths via residual and FiLM (Feature-wise Linear Modulation) fusion.
Selective Guidance: Early streams guide shallow layers (preserving spatial structure), while late streams guide deeper layers (refining semantics).
Temporal Scheduling: Guidance strength is scheduled over diffusion time, allowing structural hints to dominate early denoising steps and semantic hints to refine the image in later steps.

The training objective combines the standard diffusion loss with structural regularizers ( $L_1$ , gradient preservation, and SSIM) to ensure pixel consistency and edge preservation.

Key Contributions

Hi-DREAM Framework: A novel diffusion framework that structures fMRI conditioning into early, middle, and late visual ROI streams, mirroring the hierarchical organization of the visual cortex.
Architectural Design: The design of a ROI Adapter that converts cortical responses into compact multi-scale priors and a Depth-Matched ROI-ControlNet that injects these priors into specific U-Net depths, enabling anatomy-aware guidance without the computational overhead of full 3D volume conditioning.
Interpretability and Attribution: The framework provides a clear mapping between cortical streams and generative depths. Ablation studies demonstrate that the hierarchy-aware conditioning is not only effective but also interpretable, with different ROI streams providing complementary, inspectable contributions.

Experimental Results

The method was evaluated on the Natural Scenes Dataset (NSD) using subjects 1, 2, 5, and 7.

Quantitative Performance: Hi-DREAM achieved State-of-the-Art (SoTA) performance on high-level semantic metrics, including Inception and CLIP scores, outperforming prior methods like MindEye2, Psychometry, and MindBridge. It also retained strong low-level structural fidelity, achieving the best AlexNet(2) score and second-best PixCorr and AlexNet(5) scores.
Qualitative Performance: Reconstructions preserved object identity, scene layout, and salient visual attributes (e.g., color fidelity and shape geometry) more consistently than baseline diffusion decoders.
Ablation Studies:
- Stream Composition: Using all three streams (Early + Middle + Late) yielded the best balance of structural and semantic metrics, confirming their complementary nature.
- Hierarchy Validation: Randomly assigning ROIs to streams or reversing the hierarchy (injecting early ROIs into deep layers) significantly degraded performance, validating the importance of matching cortical hierarchy to U-Net depth.
- Module Contribution: Removing the ROI Adapter or the MHLA module resulted in substantial performance drops, proving that both the spatial priors and the cross-stream coordination are essential.
- Depth Alignment: Analysis of injected energy confirmed that the model learned to rely on Early streams for shallow layers, Middle streams for intermediate layers, and Late streams for deep layers.

Significance and Claims

The paper claims that Hi-DREAM represents a shift from treating fMRI signals as undifferentiated global embeddings to utilizing a structured, brain-inspired approach. By aligning the conditioning signal with the natural hierarchy of the visual cortex and the depth of the diffusion model, the framework achieves superior semantic reconstruction while maintaining structural integrity.

The authors emphasize that the framework offers interpretability: each conditioning stream corresponds to a defined cortical ROI group, allowing researchers to inspect how specific brain regions influence different stages of image generation. While the current implementation is subject-specific and focuses on a compact set of visual ROIs, the work establishes a foundation for more neuro-aligned, efficient, and interpretable brain-to-image decoding. Future directions identified by the authors include improving fine-grained visual details, expanding anatomical coverage to include dorsal and motion-sensitive regions, and developing cross-subject generalization strategies.