Technical Summary: Mind-Omni

1. Problem Statement

Current research in Brain-Computer Interfaces (BCIs) and neural modeling is dominated by a paradigm of specialized, single-task models. Existing approaches typically focus on isolated objectives, such as image reconstruction (decoding) or stimulus prediction (encoding), often relying on "expert models" optimized for a singular direction. This specialization curtails model versatility, neglects the potential synergistic relationships between different neural tasks, and fails to provide a unified computational framework for understanding the intrinsic mechanisms governing vision, language, and neural signals.

Furthermore, constructing a unified model faces three substantial challenges:

1. Input Heterogeneity: Inter-subject anatomical variability prevents direct integration of brain signals, and existing alignment methods often scale poorly or sacrifice information fidelity.
2. Modality Disparity: A persistent semantic gap exists between continuous brain signals (e.g., fMRI) and discrete visual/textual tokens, making direct cross-modal fusion ineffective.
3. Task Interdependence: The relationships between different encoding and decoding tasks are poorly understood. While specialized models assume task independence, it is hypothesized that shared neural representations could enable positive transfer, but this requires a unified architecture to test.

2. Methodology: Mind-Omni

To address these challenges, the authors propose Mind-Omni, the first versatile framework that unifies seven distinct encoding and decoding tasks (involving Brain, Image, and Text modalities) through a discrete diffusion paradigm. The framework consists of two primary components:

A. Standardized Neural Representation

To handle inter-subject variability, the framework registers all functional data to the MNI152 standard space. This establishes consistent structural-space dimensionality across subjects, providing a scalable foundation and mitigating the need for subject-specific modules.

B. Brain Tokenizer

A novel Brain Tokenizer transforms heterogeneous, continuous fMRI signals into standardized, discrete tokens aligned with image and text representations. This tokenizer employs a VQ-VAE-style backbone augmented with three specific alignment strategies to bridge the modality gap:

• Coarse-grained Alignment: Uses a tri-modal contrastive loss (InfoNCE) and feature distillation to align global fMRI features with CLIP image and text features in a shared semantic space.
• Fine-grained Alignment: Utilizes a cross-attention decoder to predict masked text tokens based on fMRI token sequences, optimizing for token-level semantic consistency.
• Perceptual Alignment: Ensures the reconstructed fMRI signal is semantically meaningful by enforcing that features predicted from the reconstruction match the ground-truth CLIP features of the stimulus.

C. Unified Discrete Diffusion Framework

Instead of composing separate task-specific modules, Mind-Omni leverages discrete diffusion modeling (initialized from the Muddit architecture) to create a single generative process.

• Architecture: The model extends the bi-modal Muddit backbone to a tri-modal structure (Image, Text, Brain) by adding dedicated input/output layers and symmetric Q, K, V projection layers for the brain branch.
• Training Objective: Every task is framed as a conditional masked token prediction problem. During training, target modalities are corrupted by replacing tokens with a [MASK] token according to a time-dependent cosine schedule. The model learns to restore these masked tokens given clean conditioning tokens.
• Task Unification: By strategically defining the sets of conditioning modalities ($C$) and target modalities ($T$), the single objective function unifies seven tasks:
  1. $I \to B$ (Image to Brain)
  2. $T \to B$ (Text to Brain)
  3. $I\&T \to B$ (Joint Image-Text to Brain)
  4. $B \to I$ (Brain to Image)
  5. $B \to T$ (Brain to Text)
  6. $B \to I\&T$ (Joint Brain to Image-Text)
  7. Brain Question Answering (BQA): A novel instruction-tuning task where the model generates answers based on brain signals and questions.

D. Data Curation

The authors curated a specialized Brain Question Answering (BQA) instruction-tuning dataset. Leveraging Qwen2-VL (7B) and LLaVA-Instruct-150K, they synthesized diverse training samples including detailed descriptions, concise summaries, and reasoning Q&A pairs to unlock advanced reasoning capabilities.

3. Key Contributions

1. Paradigm Shift: Mind-Omni is the first framework to unify seven distinct neural encoding and decoding tasks across brain, image, and text within a single discrete diffusion architecture, moving beyond the dominant specialized model paradigm.
2. Novel Cross-Modal Fusion: The introduction of the Brain Tokenizer bridges the gap between neural signals and visual-textual data, enabling direct, token-level interactions in a shared semantic space.
3. Demonstrated Multi-Task Synergy: The work provides empirical evidence for synergistic effects in joint neural modeling, showing that unified training can yield performance competitive with, and sometimes superior to, larger specialized models.

4. Experimental Results

The model was trained on fMRI data from 8 subjects (Natural Scenes Dataset) and evaluated on subjects 1, 2, 5, and 7.

• Holistic Performance: Mind-Omni establishes a new state-of-the-art (SOTA) among unified frameworks. While it may lag slightly behind specialized single-task models (e.g., MindEye2) on specific low-level metrics like $B \to I$ reconstruction, it maintains competitive performance across the full spectrum of seven tasks, a capability no other unified model possesses.
• Neural Decoding:
  • Visual Reconstruction: The model achieves competitive performance in both low-level (pixel correlation, SSIM) and high-level (AlexNet, CLIP) metrics.
  • Language Decoding: On detailed description and reasoning tasks, Mind-Omni demonstrates dominant SOTA performance compared to specialized models like UMBRAE and OneLLM. Notably, Mind-Omni achieves this without relying on external frozen LLMs (unlike UMBRAE and OneLLM which use Vicuna-13B or LLaMA2), suggesting a deeper understanding grounded in brain signals.
• Neural Encoding: The model outperforms multi-task baselines (BraVL, MoPoE) across all metrics (PCC, MSE, RSA) in both voxel and semantic spaces.
• Synergistic Effects:
  • Inter-modal Complementarity: Joint encoding ($I\&T \to B$) consistently outperforms single-modality encoding, capturing the brain's natural integration of visual and semantic information.
  • Inter-task Synergy: Joint decoding ($B \to I\&T$) improves both image and caption outputs compared to single-task decoding, enriching captions with visual details and suppressing artifacts.
• Computational Testbed: The framework successfully replicates known category-selective brain regions (e.g., EBA for bodies, FFA for faces, PPA for places) and probes novel concept-selective regions, validating its utility for neuroscientific exploration.

5. Significance and Claims

The authors claim that Mind-Omni offers a powerful new paradigm for neural modeling. By demonstrating that a unified, multi-task approach can achieve performance comparable to or better than specialized models, the work validates the efficacy of multi-task synergy in neural decoding.

The paper positions Mind-Omni not just as a performance benchmark, but as a computational testbed for revealing intrinsic relationships between vision, language, and neural signals. It paves the way for foundation models of neural activity, suggesting that a single architecture can effectively model the complex, distributed nature of brain cognition. The authors emphasize that their approach avoids the confounding biases of autoregressive models (which impose sequential dependencies) by utilizing the permutation-invariant nature of discrete diffusion, thereby allowing for an unbiased investigation of cross-task synergies.

The code is publicly available, and the authors intend to release processed fMRI data and instruction-tuning datasets to facilitate further research.