Technical Summary: The Capacity of Thought: Benchmarking Llama 3.2 in Semantic fMRI Neural Language Decoding

Problem Statement

The core challenge addressed is the non-invasive decoding of continuous language from functional MRI (fMRI) signals. While previous successful approaches utilize encoding models (predicting brain activity from text) to indirectly decode thoughts via search algorithms, these methods rely on older, smaller language priors (e.g., GPT-1). The authors hypothesize that the performance ceiling of these methods may stem from architectural underfitting rather than inherent biological noise. They investigate whether modern, high-capacity language models (specifically Llama-3.2-1B) can improve reconstruction accuracy and whether direct fMRI-to-language generation architectures can outperform traditional linear encoding pipelines.

Methodology

The study employs two complementary investigations on the same standardized narrative-listening fMRI dataset (8 subjects, ~16 hours of data), utilizing story-level train/test splits.

1. Improved Ridge Regression Baseline (Huth et al. Pipeline)

The authors first established a controlled empirical floor by replicating and enhancing the Huth et al. encoding-model pipeline. The core architecture uses L2-regularized ridge regression to map text features (from GPT-1 layer 9) to voxel-level BOLD responses. Decoding is performed via beam search, ranking candidate sentences against the fitted noise model and language prior.
Three specific improvements were introduced:

• Expanded Voxel Selection: Increased the selection of informative voxels from 10,000 to 15,000 via bootstrap correlation to capture broader cortical regions involved in semantic processing.
• GPT-2 Medium Proposal Model: Replaced the GPT-1 model used for generating beam-search candidates with GPT-2 Medium (345M parameters) to improve the grammatical quality of candidate words, while retaining GPT-1 strictly for the encoding features. A greedy first-subtoken proxy was implemented to handle GPT-2's byte-pair encoding vocabulary.
• GPU-Accelerated Training: Migrated the bootstrap ridge regression training from CPU (NumPy) to GPU (PyTorch) using multiprocessing, achieving a 10× speedup.

2. fMRIFlamingo: Direct fMRI-to-Language Generation

The second investigation introduces fMRIFlamingo, a direct generation framework adapting the OpenFlamingo multimodal architecture to neural decoding. The pipeline maps raw BOLD signals directly into a frozen Llama-3.2-1B backbone.

• fMRITokenizer: Raw fMRI volumes are grouped into 2,000 Regions of Interest (ROIs) via deterministic anatomical assignment. A learned linear projection maps temporal profiles to an inner dimension, preserving hemodynamic temporal structure.
• Perceiver Resampler: Compresses the ROI token sequence into a fixed number of latent vectors matching the Llama hidden dimension.
• Gated Cross-Attention: Inserted into the frozen Llama decoder layers. The gate is initialized to ensure the "vision" (brain) path is active. The model learns to mix brain-driven and text-driven attention.
• Training Strategy: To mitigate posterior collapse (where the model ignores noisy fMRI data in favor of text priors), 50% of input prompt tokens are masked with <PAD> during training. Optimization uses cross-entropy loss and a contrastive ranking loss.
• Inference: Operates in a per-window regime, generating one word per 2-second TR window via 3-beam search, distinct from the continuous narrative decoding of the ridge pipeline.

Key Results

Improved Ridge Baseline

On subject UTS03 across three held-out narratives, the improved pipeline achieved:

• METEOR: 0.149 (mean), representing an 11% relative gain over the 10K-voxel replication baseline (0.134).
• BLEU-1: 0.200 (mean).
• BERTScore: 0.814.
  These results are competitive with the original scores reported by Huth et al. on their specific test narratives, establishing a stronger linear decoding baseline.

fMRIFlamingo Performance

• Apparent Success: On a 1-in-100 ranking task (selecting the correct next word from 99 distractors), the model achieved 42.86% Top-1 accuracy, significantly above the 1% random chance.
• Blind Control Failure: A critical ablation study zeroed out all fMRI inputs. The ranking accuracy remained virtually unchanged (~43%), and story-level generation metrics (WER, BLEU-1, METEOR) were statistically identical to the full-input condition.
• Language Prior Dominance: The model's output distribution in the zero-fMRI condition was nearly identical to the held-out condition, dominated by frequent function words. When preceding text context was removed entirely ("Strict Telepathy" condition), accuracy dropped to chance (~0%).
• Conclusion: The apparent decoding success is driven primarily by the frozen language prior and linguistic context rather than the neural input. The model incorporates the fMRI signal but fails to learn representations reliable enough to improve upon the text-only prior.

Significance and Claims

The paper makes two primary claims regarding the state of neural language decoding:

1. Refinement of Linear Baselines: Incremental engineering improvements to traditional encoding pipelines (voxel expansion, better proposal models, and hardware acceleration) can yield significant performance gains (11% METEOR improvement), establishing a robust baseline for future comparisons.
2. The "Language Prior Illusion": The integration of high-capacity, frozen Large Language Models (LLMs) into neural decoding does not inherently improve performance. Instead, these models can actively obscure failures. Because LLMs possess strong statistical priors, they can generate fluent, plausible outputs even when neural inputs are absent or uninformative.
   • Methodological Imperative: The authors argue that without rigorous blind-control ablation experiments (zeroing neural inputs), claims of successful neural decoding using large models are likely invalid. The 42% ranking accuracy observed in fMRIFlamingo is presented not as a breakthrough in decoding, but as a demonstration of how easily strong language priors can mimic decoding success.

The study concludes that for small language models to decode continuous semantic meaning from brain activity, they likely require finetuning on large, aligned datasets of text and brain activity (which currently do not exist in open source) and more effective architectural mechanisms to isolate relevant brain regions.