Technical Summary: Real-time Reconstruction of Human Visual Perception from fMRI

Problem Statement

While functional magnetic resonance imaging (fMRI) has enabled significant advances in decoding fine-grained cognitive states and reconstructing visual stimuli, a critical gap exists between state-of-the-art offline decoding methods and real-time applications. Advanced decoding pipelines, such as those utilizing foundation models, are computationally intensive, often requiring hours to days of processing. Consequently, they do not fit within the temporal envelope of real-time analysis, which necessitates processing data within seconds without access to future session data. This limitation hinders the deployment of sophisticated brain-computer interfaces (BCIs) and closed-loop neurofeedback systems that could leverage fine-grained reconstruction for scientific discovery and clinical treatment. The primary challenge is adapting these heavy computational pipelines to operate under the strict latency constraints of real-time fMRI while accounting for the intrinsic hemodynamic lag of the BOLD signal.

Methodology

The authors present a real-time compatible adaptation of MindEye2, a large-scale (>700M parameters) machine learning architecture for visual decoding. The implementation relies on RT-Cloud, an open-source, scalable cloud-based platform designed for streaming and analyzing fMRI data.

Data Collection and Experimental Design

• Scanner: 3 Tesla (3T) Siemens Prisma scanner, a widely available clinical platform, rather than the less common 7T systems used in prior high-resolution work.
• Participants: Data was collected from a single participant (one of the authors) and validated using a held-out subject from the Natural Scenes Dataset (NSD).
• Stimuli: Natural images (locations, animals, people) primarily from NSD. The experimental design utilized a rapid event-related paradigm with 531 unique images presented in 11 functional runs.
• Protocol: The participant viewed images for 3 seconds with 1-second inter-stimulus intervals. A familiarity detection task ("new" vs. "old") was used to maintain attention.

Pipeline Modifications for Real-Time Processing

To transition from offline to real-time analysis, the authors modified the standard MindEye2 pipeline:

1. Preprocessing:
   • Streaming: 3D brain volumes are streamed from the scanner every TR (1500 ms).
   • Registration & Motion Correction: Affine registration and motion correction (FSL MCFLIRT) are applied in real-time to align volumes to a reference from the training session.
   • Hemodynamic Lag Handling: To capture the peak BOLD response, inference is delayed. The authors tested three delay strategies:
     • Fast: ~7.9 seconds (2 trials) post-stimulus.
     • Slow: ~29 seconds (7 trials) post-stimulus.
     • End-of-run: ~5 minutes (end of the functional run).
   • GLM Estimation: A General Linear Model (GLM) using the Least Squares-Separate (LSS) approach is fitted to extract single-trial beta estimates, accounting for the extended BOLD response and overlapping stimuli.
2. Model Architecture:
   • A condensed version of MindEye2 was used, omitting low-level submodules and text refinement steps, and reducing the shared-subject latent dimensionality from 4096 to 1024 to speed up inference.
   • Input: Single-trial betas from a "reliability mask" (voxels with high response consistency) are mapped to a shared-subject latent space via ridge regression.
   • Output: The latent space is mapped to CLIP image embeddings.
     • Reconstruction: Uses a diffusion prior and a frozen unCLIP model (Stable Diffusion XL) to generate images.
     • Retrieval: Uses an MLP projector to map to CLIP space, followed by nearest-neighbor search against a pool of candidate images.
3. Training Strategy:
   • Pretraining: The model was pretrained on data from 7 NSD subjects (7T data).
   • Fine-tuning: The model was fine-tuned on approximately 1 hour of 3T data from a new participant (using averaged betas from the first scan session).
   • Evaluation: Tested on a subsequent session using single-trial betas from the first presentation of test images.

Key Results

The study demonstrates that fine-grained visual decoding is feasible in real-time on a standard 3T scanner.

• Latency vs. Performance:
  • Fast Real-Time: Retrieval is achieved in 9.5 seconds and reconstruction in 14.5 seconds post-stimulus.
  • Slow Real-Time: Retrieval takes ~36 seconds.
  • End-of-Run: Takes ~2.7 minutes.
  • Offline: Takes >24 hours.
• Decoding Accuracy:
  • Retrieval: Even in the "fast" real-time setting (9.5s delay), the system achieved 36% top-1 accuracy on a pool of 50 images (chance = 2%). At a pool size of 2, accuracy reached 90.6%.
  • Reconstruction: Qualitative results show recognizable reconstructions of natural scenes, though with lower fidelity compared to offline 7T reconstructions.
  • Comparison: Offline 3T performance (fine-tuned) approaches offline 7T performance, validating the transferability of the model across field strengths.
• Trade-offs: There is a positive correlation between stimulus delay and decoding performance. Performance gains diminish after ~30 seconds, suggesting an optimal trade-off point between speed and accuracy.
• Generalization: The pipeline successfully generalized from 7T pretraining to 3T fine-tuning and testing, requiring only ~1 hour of new participant data.

Key Contributions

1. First Real-Time Visual Reconstruction: This work presents the first pipeline to achieve real-time reconstruction of visual perception from fMRI at single-trial resolution.
2. 3T Feasibility: Demonstrates that high-performance decoding is possible on widely available 3T scanners, not just specialized 7T systems.
3. Minimal Training Data: Shows that reliable real-time decoding can be achieved with minimal training data (~1 hour) from a new participant.
4. Quantified Trade-offs: Systematically documents the performance-latency trade-offs when transitioning from offline to real-time analysis, providing a benchmark for future BCI designs.
5. Open Science: All code, models, and data are made publicly available to encourage collaboration and downstream applications.

Significance and Claims

The authors position this work as a proof-of-concept demonstrating the feasibility of deploying computationally intensive, state-of-the-art fMRI decoding pipelines in real-time.

• BCI and Neurofeedback: The primary significance lies in enabling novel closed-loop experiments. The authors suggest that while full image reconstruction may be too slow for some applications, the high retrieval accuracy at short latencies (e.g., distinguishing between two images) could support neurofeedback paradigms. For instance, participants could be trained to modulate their neural representations to be closer or further apart in the latent space, potentially aiding in treating conditions like depression by altering perception of negative stimuli.
• Scientific Discovery: The ability to decode thoughts in near real-time opens new avenues for studying fundamental learning mechanisms and cognitive processes as they happen, rather than in post-hoc analysis.
• Modest Claims: The authors explicitly note that while the results are above chance and promising, they do not yet match the fidelity of offline 7T reconstructions. They emphasize that the current latency (10–30 seconds) is suitable for learning based on existing neurofeedback literature, but further optimization is needed for more immediate feedback loops. The work serves as a foundational step toward integrating powerful AI algorithms into real-time clinical and research settings.