Technical Summary: Feature Visualization Recovers Known Cortical Selectivity from TRIBE v2

Problem Statement

Brain encoder models, which predict cortical fMRI responses from the internal activations of pretrained vision and language networks, are currently evaluated almost exclusively by held-out prediction accuracy (e.g., $R^2$). While this metric is effective for training, it is insufficient for interpretation. High accuracy confirms that an encoder fits the data but does not verify whether the model has internalized the functional organization of the brain. Two encoders with identical held-out accuracy may make systematically different predictions about the input-side preimage of a cortical region. Consequently, there is a need for a complementary interpretability technique that can qualitatively assess whether an encoder has recovered the specific functional selectivity of brain regions (e.g., edges for V1, faces for FFA, motion for MT).

Methodology

The authors propose feature visualization as a probe for brain encoders. This technique involves performing gradient ascent on the encoder's predicted activation for a target Region of Interest (ROI) to synthesize an input stimulus that maximally drives that region.

Experimental Setup:

• Models: The study applies this probe to TRIBE v2, a brain encoder composed with V-JEPA 2 (a 40-layer ViT-G visual backbone pretrained on video with a joint-embedding predictive objective). Both models are frozen; only the input image parameters are optimized.
• Target ROIs: Seven cortical regions spanning the visual hierarchy were selected based on the HCP-MMP1 parcellation: V1, V2, V3, V4, MT (middle temporal area), FFA (fusiform face area), and PPA (parahippocampal place area).
• Optimization Process:
  • Input Parameterization: Images are parameterized as a real-valued 2-D Fourier spectrum on decorrelated color channels, mapped to the $[0, 1]$ pixel range via inverse FFT and sigmoid.
  • Static Optimization: Despite V-JEPA 2 being a video model, the optimization synthesizes a single static frame tiled to 64 identical frames. This reduces computational cost while testing if static cues suffice to drive motion-selective regions.
  • Loss Function: The objective is a "global lift" loss: $L = -\frac{1}{|R|}\sum_{v \in R} \hat{y}_v + \beta \cdot \frac{1}{|\bar{R}|}\sum_{v \in \bar{R}} \hat{y}_v + \lambda_{fft} \|S\|^2$. This maximizes the mean activation of the target ROI while subtracting the mean activation of the rest of the cortex (to encourage selectivity over generic drive) and includes a spectral energy regularizer to suppress pathological high-frequency textures.
  • Hyperparameters: Optimization runs for 3,000 gradient steps. Five random-seed restarts are performed per ROI to assess stability.

Key Results

1. Qualitative Recovery of Cortical Selectivity

The synthesized stimuli demonstrate a clear recovery of known functional anatomy:

• V1 $\to$ V4 Progression: There is a visible progression of increasing spatial scale and feature complexity. V1 produces dense fields of small-scale oriented edges (Gabor-like); V2 and V3 show larger contours and junctions; V4 exhibits organized mid-scale curves.
• MT (Motion Area): Despite optimizing only static images, all five restarts for MT produce sharp radial or diagonal streak patterns resembling "frozen optic flow" or long-exposure photography. This indicates the encoder has learned to associate static motion cues (streaks, elongated orientations) with the MT region.
• FFA (Face Area): The probe generates varied but recognizable face-like features (eye regions, nose ridges, mouth outlines) across restarts.
• PPA (Place Area): The stimuli produce consistent rectilinear textures (parallel, predominantly horizontal and diagonal lines). While less category-specific than FFA, this aligns with the region's sensitivity to scene layouts and architecture.

2. Quantitative Selectivity and "Super-Stimuli"

• Activation Lift: The optimized stimuli drive their target ROIs significantly above a random-noise baseline. Notably, the FFA-optimized stimulus drives the predicted FFA activation $\sim4\times$ higher than a natural face photograph (+0.339 vs. +0.080). This confirms the "super-stimulus" phenomenon, where gradient ascent finds adversarial patterns that exceed natural distribution limits.
• Cross-ROI Structure: The activation matrix reveals expected anatomical relationships. Early visual areas (V1–V3) show strong cross-activation. FFA and PPA are mutually suppressive. V4 acts as a bridge, driving FFA more strongly than early visual areas.
• MT Baseline: MT shows the largest activation lift (+0.70), partly because its random-noise baseline is strongly negative (indicating deactivation for non-motion stimuli).

3. Stability and Limitations

• Stability: The qualitative signatures (e.g., face features for FFA, streaks for MT) are stable across all five random seeds, with variation restricted to specific geometry rather than category.
• Reproducibility: The authors note that CUDA fp16 non-determinism causes trajectory divergence, meaning the highest-activation restart is not always the most visually interpretable (e.g., the clearest face image was not the activation maximum).
• PPA Ambiguity: The authors explicitly state they are less comfortable interpreting PPA results, as the stimuli look like line patterns rather than recognizable scenes, highlighting a limitation in the current probe's ability to resolve category-level labels for all regions.

Significance and Claims

The paper claims that feature visualization serves as a necessary, complementary qualitative test to prediction accuracy for brain encoders.

• Internalization vs. Fitting: The results suggest TRIBE v2 has not merely fit the training fMRI time series but has internalized the functional organization of the visual hierarchy. The ability to invert the encoder to recover specific tuning signatures (edges, motion, faces) from a static optimization pipeline is presented as evidence of this internalization.
• Methodological Reach: The recovery of "implied motion" in MT from static inputs is highlighted as a key demonstration of the method's capability, showing that the encoder's motion readout can be inverted into recognizable motion-evoking imagery even without temporal optimization.
• General Applicability: The probe is described as simple, differentiable, and applicable to any brain encoder with a differentiable backbone, offering a "cheap, default qualitative test" to distinguish encoders that have recovered neuroscience principles from those that simply predict data.

The authors remain modest, acknowledging that optimized stimuli are extrema of the response surface and may contain adversarial features. They emphasize that while the probe reveals what the encoder thinks a region looks for, it does not fully resolve whether these features map perfectly to biological reality without further validation (e.g., video-axis optimization or human observer studies).