Technical Summary: CHASMBrain

Problem Statement

The fundamental challenge addressed in this work is understanding the correspondence between deep visual representations and the hierarchical organization of the human visual system. While modern vision models excel at image recognition, their alignment with the functional organization of the human visual cortex remains an open question. Specifically, existing encoding models often struggle to simultaneously capture:

1. Local spatial details critical for early visual areas (retinotopic regions like V1/V2).
2. Global semantic context essential for higher-order cortical areas.
3. High intrinsic noise in fMRI signals, which complicates direct voxel-level regression.

Current approaches often rely on hand-crafted features or monolithic deep networks that lack biological interpretability or fail to disentangle spatial location from feature tuning.

Methodology: CHASMBrain

The authors propose CHASMBrain, a novel hierarchical two-stage framework for image-to-fMRI encoding. The architecture is motivated by the functional organization of the visual cortex, employing a coarse-to-fine strategy and a dual-stream Mamba design.

1. Input Representation

The model utilizes a pre-trained DINOv2 encoder (ViT-B/14) to extract visual features:

• CLS Token: A single vector summarizing global image semantics.
• Patch Tokens: A sequence of 256 vectors preserving local spatial structure.

2. Stage 1: Coarse Prediction (Dual-Stream Mamba)

The first stage predicts denoised activations at the Region of Interest (ROI) level rather than individual voxels to mitigate noise.

• Dual-Stream Architecture:
  • Global Branch: Processes the CLS token using a Mamba block to model high-level semantics.
  • Local Branch: Processes patch tokens using a deeper stack of Mamba blocks to model fine-grained spatial structure.
• Adaptive Fusion: A learnable gate dynamically fuses the outputs of the two branches ($\hat{r} = \alpha \cdot r_g + (1-\alpha) \cdot r_l$), allowing the model to emphasize global context for simple scenes or local details for complex ones.
• ROI Aggregation: Voxels are clustered into non-overlapping groups. Instead of simple averaging, the model aggregates the top-$p\%$ (set to 70%) of activations within each cluster to preserve informative peaks while suppressing noise.
• Training Objective: Combines Mean Squared Error (MSE), Pearson correlation alignment, and a contrastive InfoNCE loss to align image features with brain responses.

3. Stage 2: Fine Prediction (Mamba-VAE)

The second stage refines the coarse ROI predictions into full voxel-level predictions.

• Input Fusion: Combines the Stage 1 ROI predictions with the original visual features (pooled patch features and CLS token).
• Mamba-VAE:
  • Encoder: A stack of Mamba blocks encodes the fused sequence.
  • Variational Latent Space: A VAE head models trial-to-trial variability in fMRI responses by learning a latent distribution ($\mu, \sigma$). During inference, the mean ($\mu$) is used for deterministic prediction.
• Dual Decoding:
  • Holistic Voxel Decoder: Predicts the full voxel vector jointly from global context using learnable query tokens.
  • ROI Residual Decoder: Predicts local residuals relative to the Stage 1 coarse values, focusing on fine corrections.
• Amplitude Recovery: A learnable scale-recovery layer adjusts the output magnitude based on stimulus complexity.

Key Contributions

1. Hierarchical Two-Stage Framework: CHASMBrain achieves a Pearson correlation of 0.429 and an MSE of 0.261 on the Natural Scenes Dataset (NSD), outperforming recent deep models (SynBrain, MindSimulator) and strong linear baselines (Ridge Regression, DINOv2 linear probes).
2. Sequential Coarse-to-Fine Mechanism: By separating robust semantic alignment at the ROI level from fine-grained voxel refinement via a Mamba-VAE, the model effectively addresses the high intrinsic noise of fMRI signals.
3. Causal Stream Specialization: Through branch-ablation and input-routing experiments, the authors demonstrate an asymmetric specialization:
   • The Patch (Local) stream is causally locked to early visual cortex (retinotopic regions). Removing it disproportionately harms early region predictions.
   • The CLS (Global) stream contributes broader semantic context to higher-order areas.
   • Swapping these streams causes a significant asymmetric drop in performance (1.9× relative degradation in early regions), confirming the correspondence is causal rather than merely correlational.
4. Cross-Subject Generalizability: The learned backbone generalizes across individuals. When trained on source subjects and frozen, with only a lightweight per-subject head fine-tuned, the model achieves near within-subject performance on held-out subjects, suggesting the representation is subject-agnostic.

Results

• Quantitative Performance: On NSD (subjects 1, 2, 5, 7), CHASMBrain reduces MSE by 26.9% compared to the strongest DINOv2 linear probe (Layer 12) and significantly outperforms prior works like SynBrain and MindSimulator.
• Qualitative Analysis: Predicted fMRI maps preserve sharp regional boundaries and localized peaks, closely mirroring ground-truth neural responses. Downstream image reconstruction (via MindEye2) demonstrates robust semantic retention, successfully reconstructing coherent structures of rigid objects better than baselines.
• Ablation Studies:
  • The Dual-Mamba design outperforms single-stream Mamba and Transformers while using fewer parameters and FLOPs.
  • The VAE-based refinement strategy significantly outperforms diffusion-based refinement, which fails due to conflating artificial noise with biological noise.
  • An optimal clustering granularity of 30 voxels/cluster was identified, balancing noise reduction in Stage 1 with spatial resolution for Stage 2.

Significance

The paper claims that CHASMBrain bridges the gap between artificial visual representations and human neural activity by incorporating cortical priors. By explicitly separating spatial location parameters from feature-weight parameters, the model enables direct inspection of receptive-field structures. The causal evidence of stream specialization suggests that the model captures a shared, biologically plausible visual representation that mirrors the "where" (dorsal/early) and "what" (ventral/higher) pathways of the human brain. Furthermore, the cross-subject generalization indicates that the model learns fundamental visual principles rather than overfitting to subject-specific noise.

Limitations

The authors acknowledge several limitations:

• The predictive upper bound is constrained by the physiological noise ceiling of fMRI ($r \approx 0.60$).
• The pipeline relies on a frozen DINOv2 encoder biased toward static, object-centric images, potentially lacking temporal dynamics.
• Cross-dataset evaluation is not yet performed due to differing acquisition protocols.
• Downstream visual reconstruction is bottlenecked by the generative priors of the fMRI-to-image decoder.