💻 computer science

Seeing Through the Brain: New Insights from Decoding Visual Stimuli with fMRI

The paper proposes PRISM, a novel framework that projects fMRI signals into a structured text space and employs object-centric diffusion with attribute-relationship search to achieve state-of-the-art visual stimulus reconstruction by better aligning with the compositional nature of neural activity.

Original authors: Zheng Huang, Enpei Zhang, Weikang Qiu, Yinghao Cai, Carl Yang, Elynn Chen, Xiang Zhang, Rex Ying, Dawei Zhou, Yujun Yan

Published 2026-05-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zheng Huang, Enpei Zhang, Weikang Qiu, Yinghao Cai, Carl Yang, Elynn Chen, Xiang Zhang, Rex Ying, Dawei Zhou, Yujun Yan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine your brain is a busy library where every time you see an image, it doesn't just "store" the picture. Instead, it writes a very specific, complex story about what it sees. For years, scientists have tried to read these stories (from fMRI brain scans) and turn them back into the original pictures you saw.

The paper you're asking about, PRISM, is like a new, smarter translator that finally cracked the code on how to do this better. Here is the simple breakdown of what they found and built:

1. The Big Surprise: The Brain Speaks "Text," Not "Pictures"

Most previous attempts to rebuild images from brain scans tried to translate the brain's signal directly into a "picture language" (like a digital photo file). They assumed that because you saw a picture, your brain must be storing it as a picture.

The PRISM team discovered this was wrong.
They found that the brain's activity actually looks much more like text (like the sentences in a book) than it does like a photo or a video.

The Analogy: Imagine trying to translate a foreign language. Previous methods tried to translate the brain's "language" into another visual language (like translating a French movie directly into a German movie). PRISM realized the brain is actually speaking English (text). So, the best way to understand it is to translate it into English first, and then use that English to describe the picture.

2. The Problem with "Blurry" Descriptions

Even when scientists used text, they often made a mistake. They would write a single, long sentence describing the whole image, like: "A gray tiger-striped cat is sitting on a bench."
When a computer tries to draw this, it often gets confused. It might draw a "gray tiger" instead of a "cat," or mix up the colors. This is called an "attribute binding error"—the computer knows the words but forgets which word belongs to which object.

The Human Brain doesn't do this.
When you look at a scene, your brain doesn't see one giant blob. It sees distinct items: "There is a cat. It is gray. It has stripes. It is on a bench." It builds the scene piece by piece.

3. The Solution: PRISM (The "Lego" Builder)

The authors built a new system called PRISM (Projects fMRI sIgnals into a Structured text space). Think of it as a master builder who uses Lego bricks instead of a single giant mold.

Here is how PRISM works, step-by-step:

Step 1: The Translator (fMRI to Text)
The system takes the brain scan and translates it into a structured text list. Instead of one long paragraph, it breaks the image down into specific parts:
- Object 1: A car.
- Object 2: An elephant.
- Relationship: The car is following the elephant.
- Attributes: The car is white; the elephant is gray.
Step 2: The Smart Search (Finding the Right Words)
The system uses a "search engine" to figure out exactly which words matter most. It tested thousands of different ways to describe things and found that spatial words (like "left," "right," "on top of," "next to") are the most important for matching the brain's activity. It's like realizing that to describe a room to someone blindfolded, telling them where things are is more important than telling them what color the curtains are.
Step 3: The Lego Builder (Object-Centric Generation)
Instead of asking an AI to draw the whole picture at once, PRISM asks it to draw the car first, then the elephant, and then puts them together in the right spots based on the text description.
- Why this works: It stops the computer from getting confused. It ensures the car stays a car and the elephant stays an elephant, and they don't accidentally merge into a "car-elephant."

4. The Results

When they tested this on real people looking at images, PRISM did a much better job than previous methods.

The Score: It reduced the "perceptual loss" (how much the new image looks different from the original) by about 6%.
The Proof: If you showed the reconstructed images to a smart AI and asked, "What is in this picture?", the AI could answer correctly much more often with PRISM's images than with older methods.

Summary

The paper claims that to read the brain's visual thoughts, we shouldn't try to force the brain to speak in "pixels." Instead, we should listen to it as if it were writing a structured story about objects and their locations. By translating brain scans into this specific type of "object-based text" and then building the image piece-by-piece, we can see what the person was looking at with much greater clarity.

Note: The paper focuses strictly on the technical method of reconstructing static images from brain scans. It does not claim to have clinical applications, medical uses, or the ability to read thoughts in real-time for communication at this stage.

Technical Summary: PRISM – Seeing Through the Brain via Structured Text

Problem Statement

Reconstructing visual stimuli from functional Magnetic Resonance Imaging (fMRI) signals is a central challenge in neuroscience and machine learning. Current approaches typically involve two stages: mapping fMRI signals into a latent space and then using a pre-trained generative model (often a diffusion model) to reconstruct images. However, two critical limitations persist:

Latent Space Misalignment: Prior work largely assumes that the latent space must match the modality of the stimulus (i.e., using vision-based representations). It remains unclear whether vision-based spaces are optimal or if other spaces, such as text, might better align with neural activity.
Compositional Mismatch: Existing methods often rely on unified hidden representations that conflate objects with their attributes. This leads to "attribute binding" errors (e.g., generating a "grey tiger" instead of a "grey cat") and fails to capture the object-centric, compositional nature of human visual processing.

Methodology: PRISM

The authors propose PRISM (Projects fMRI sIgnals into a Structured text space as an interMediate representation), a framework that addresses these limitations through three core components:

1. Text as the Optimal Latent Space

The authors first investigate the alignment between fMRI signals and various representation spaces: language model (LM) text space, vision-language model (VLM) joint space, and vision-only space.

Finding: Using Centered Kernel Alignment (CKA), Canonical Correlation Analysis (CCA), and Generalization Gap metrics, the study demonstrates that fMRI signals align more closely with the text space of language models than with vision-based or joint text-image spaces.
Implication: This motivates the use of pure text embeddings as the intermediate latent space for reconstruction, challenging the assumption that vision-based representations are essential.

2. Attribute/Relationship Search Module

To address the compositional nature of visual stimuli, PRISM does not use generic image captions. Instead, it employs a module to automatically discover the specific attributes and relationships that best align with neural activity.

Process: The module frames keyword selection as a prompt optimization problem. It uses an $\epsilon$ -greedy search strategy with a Large Language Model (LLM) to iteratively generate and evaluate candidate keywords (e.g., "Spatial Layout," "Relative Position").
Objective: The search maximizes the similarity between images generated from VLM descriptions (using the candidate keyword) and the ground truth images, subject to a constraint that the CKA similarity between the fMRI data and the resulting text embeddings exceeds a threshold.
Outcome: The search converges on spatially oriented keywords (e.g., "Spatial Layout"), which are then used to generate structured, object-centric descriptions for training.

3. Object-Centric Diffusion and Encoder Training

The framework consists of an encoder and a specialized generative process:

Encoder Training: An encoder maps fMRI signals into the LM text space. It uses Multi-Layer Perceptrons (MLPs) to independently encode features for each object, which are then concatenated and passed to a fine-tuned language model (T5) to predict structured descriptions.
Object-Centric Generation: During inference, the predicted structured descriptions guide a pre-trained diffusion model. Unlike standard text-to-image generation, PRISM adapts the diffusion process to generate images by composing individual objects.
- Each object's description is embedded into a conditioning matrix.
- The model generates latent representations for each object independently.
- These representations are spatially resized and concatenated based on predicted locations.
- A weighted sum blends the object latents with a global context latent to ensure seamless boundaries.

Key Contributions

Novel Empirical Findings: The paper is the first to demonstrate that accurate visual reconstruction can be achieved without image-based latent representations, showing that LM text space provides superior alignment with fMRI signals compared to vision or VLM spaces.
Structured Representation: The authors show that adapting both the text space and the generative model to capture the compositional and relational nature of visual stimuli (objects, attributes, relationships) significantly improves reconstruction quality.
PRISM Framework: A new architecture that integrates an attribute/relationship search module for automatic prompt optimization and an object-centric diffusion module to mitigate attribute binding errors.

Experimental Results

The framework was evaluated on three real-world fMRI datasets: NSD, BOLD5000, and GOD.

Performance: PRISM outperforms state-of-the-art baselines (including MindEye2, MindBridge, and Takagi & Nishimoto) across all metrics (PixCorr, SSIM, LPIPS, CLIP, and Inception V3).
Key Metric: The method achieves up to a 6% reduction in perceptual loss (LPIPS) compared to existing methods, indicating higher perceptual similarity to the original images.
Semantic Consistency: In a Question Answering (QA) task using reconstructed images, PRISM achieved 60.54% accuracy, significantly outperforming other methods, confirming that the reconstructions are semantically meaningful.
Ablation Studies: Removing the object-centric diffusion module or the attribute/relationship search module consistently degraded performance, validating the necessity of both components.

Significance and Claims

The paper claims that its primary significance lies in redefining the intermediate representation for brain decoding. By establishing that text space is more brain-aligned than vision space and by explicitly modeling the compositional structure of visual perception, PRISM bridges the gap between neural activity and image generation more effectively than previous holistic approaches. The results highlight that structured text serves as a powerful bridge, allowing generative models to reconstruct visual stimuli with higher fidelity and fewer semantic errors.

The authors note that while the optimal keywords found (e.g., "Spatial Layout") align with neuroscientific findings regarding spatial memory regions (such as the Presubiculum), further validation from the neuroscience community is required. The work is presented as a step toward better understanding human visual representation and improving brain-computer interfaces.