💬 NLP

fMRI-LM: Towards a Universal Foundation Model for Language-Aligned fMRI Understanding

The paper introduces fMRI-LM, a universal foundation model that bridges functional MRI and language through a three-stage framework involving neural tokenization, joint LLM adaptation, and multi-task instruction tuning to enable scalable, language-aligned semantic understanding of brain activity.

Original authors: Yuxiang Wei, Yanteng Zhang, Xi Xiao, Chengxuan Qian, Tianyang Wang, Vince D. Calhoun

Published 2026-04-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuxiang Wei, Yanteng Zhang, Xi Xiao, Chengxuan Qian, Tianyang Wang, Vince D. Calhoun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine your brain is a massive, bustling city. Every time you think, feel, or move, different neighborhoods (brain regions) light up and talk to each other. For decades, scientists have used a special camera called an fMRI to take "photos" of this city in action. But here's the problem: these photos are just a chaotic mess of numbers and colors. They are like a raw, unedited security feed that only a handful of experts can decipher.

Enter fMRI-LM, a new AI model that acts like a universal translator between the language of the brain and the language of humans.

Here is how it works, broken down into simple steps:

1. The Problem: The Brain Speaks a Secret Code

Currently, if a doctor wants to know if a patient has Alzheimer's or just needs to know their age based on a brain scan, they have to build a tiny, specialized robot for just that one job. It's like having a different key for every single door in a building. These robots are good at their specific job but can't talk to each other, and they need thousands of examples to learn.

2. The Solution: Teaching the Brain to "Speak" English

The researchers behind fMRI-LM had a brilliant idea: What if we taught the brain to speak the same language as a chatbot?

They built a three-stage system to make this happen:

Stage 1: The "Neural Translator" (The Dictionary)

First, they needed a way to turn the messy brain scan numbers into something a language model could understand.

The Analogy: Imagine the brain scan is a complex, abstract painting. The researchers built a "translator" that looks at the painting and turns it into a set of discrete words (tokens).
How they did it: Since there are no natural "captions" for brain scans (nobody writes a sentence saying "This brain is thinking about a cat"), they invented a structured description system. They took scientific measurements (like how connected different brain areas are) and turned them into readable sentences.
- Example: Instead of just a number, the AI learns to read: "The connection between the visual area and the attention area is slightly stronger than average."
- This creates a dictionary that translates "Brain-ese" into "English."

Stage 2: The "Brain-Reading Book Club" (Training)

Now that they have a dictionary, they took a powerful, pre-trained language model (like a super-smart chatbot that already knows English) and taught it to read these new "brain sentences."

The Analogy: Imagine teaching a human to read a new language by showing them a book where the pictures are brain scans and the text is the description.
The AI learns to predict what comes next. If it sees a brain scan from a few seconds ago, it can guess what the brain will do next. It also learns to answer questions like, "Based on this scan, is this person male or female?"

Stage 3: The "Swiss Army Knife" (Instruction Tuning)

Finally, they taught the AI to handle many different tasks at once, just like a human who can read a medical report, guess someone's age, and diagnose a disease all in the same conversation.

The Analogy: Instead of building a new robot for every task, they gave this one AI a Swiss Army Knife. You can ask it:
- "Is this person healthy?"
- "What is their age?"
- "Describe this person's cognitive state."
- And it answers in natural language.

Why is this a Big Deal?

It's Universal: Before this, you needed a different model for every disease or age group. fMRI-LM is a foundation model, meaning it learns the general "grammar" of the brain. Once it learns that, it can adapt to new tasks with very little extra training (like learning a new dialect).
It's Efficient: You don't need millions of labeled brain scans to teach it new tricks. It can learn from just a few examples (a "few-shot" learner), saving time and money.
It's Interpretable: Because the AI speaks English, doctors can ask it why it made a decision. Instead of a black box saying "Disease Detected," it can say, "I detected Alzheimer's because the connection between the memory center and the language center is weaker than usual."

The Bottom Line

Think of fMRI-LM as the Rosetta Stone for the human brain. It bridges the gap between the silent, electrical signals of our neurons and the words we use to describe our thoughts and health. By turning brain scans into a language we can understand, it opens the door to faster diagnoses, better personalized medicine, and a deeper understanding of how our minds work.

1. Problem Statement

Despite the success of Large Language Models (LLMs) and Multimodal LLMs (MLLMs) in unifying reasoning across images, audio, and video, extending these capabilities to functional MRI (fMRI) remains largely unexplored. Existing approaches face several critical limitations:

Task-Specific Tuning: Traditional deep learning models for fMRI (e.g., CNNs, GNNs) require task-specific tuning and labeled data, limiting scalability and cross-study generalization.
Lack of Semantic Grounding: Recent fMRI foundation models (e.g., BrainLM, Brain-JEPA) rely on neural-only objectives (masked prediction, contrastive learning) and lack alignment with language, preventing high-level semantic understanding.
Data Scarcity: There is a fundamental absence of natural fMRI–text pairs (unlike image-caption pairs in vision), making it difficult to train models that can describe brain activity linguistically.
Limited Reasoning: Existing neural-to-text works often rely on fixed single-question templates, failing to leverage the generative and reasoning capabilities of modern LLMs.

The core challenge is to create a universal foundation model that can understand resting-state and task-independent neural patterns, align them with language, and perform diverse downstream tasks (diagnosis, phenotype prediction) without extensive retraining.

2. Methodology: The fMRI-LM Framework

The authors propose fMRI-LM, a three-stage framework designed to bridge fMRI and language. The model is pretrained on over 50,000 fMRI scans (primarily from the UK Biobank and ABCD datasets).

Stage 1: fMRI Tokenizer Training (Neural Tokenization)

The goal is to map continuous fMRI signals into discrete tokens that share a representational geometry with the text embedding space of a pretrained LLM.

Input Processing: Raw 4D fMRI data is parcellated into 450 Regions of Interest (ROIs) using the Schaefer-400 (cortical) and Tian-Scale III (subcortical) atlases.
Architecture: A Transformer-based encoder followed by a Vector Quantizer (VQ) converts the continuous latent features into discrete tokens.
Alignment Strategy: To align fMRI tokens with the frozen text space of an LLM (e.g., GPT-2), the tokenizer is trained using three objectives:
1. Reconstruction Loss: Ensures the tokenizer preserves fMRI structural information via a lightweight decoder.
2. Domain-Adversarial Loss: Uses a Gradient Reversal Layer (GRL) to make fMRI embeddings indistinguishable from text embeddings in the LLM's space.
3. Contrastive Loss: Aligns paired fMRI data with synthetic text descriptors (see below) to bridge the modality gap.

Stage 2: LLM Fine-Tuning (Temporal & Linguistic Modeling)

A pretrained LLM is adapted to model the discrete fMRI tokens and generate text.

Training Paradigms:
- fMRI-to-fMRI (F2F): Temporal next-step prediction (predicting $t+1$ tokens based on $t$ ) to capture neural dynamics.
- fMRI-to-Text (F2T): Generating descriptive text conditioned on fMRI tokens.
- Text-to-Text (T2T): Standard language modeling to preserve the LLM's original linguistic capabilities.
Key Innovation: The model learns to treat brain activity as a sequence that can be temporally predicted and linguistically described.

Stage 3: Multi-Task, Multi-Paradigm Instruction Tuning

To enable high-level semantic understanding, the model undergoes instruction tuning on diverse neuroscience and clinical tasks.

Synthetic Corpus Construction: Since natural fMRI-text pairs do not exist, the authors constructed a structured descriptive corpus. They extracted four types of imaging-based features (Functional Connectivity, Functional Gradients, Graph Metrics, ICA components) and converted them into standardized, caption-like text descriptions using templates and LLM refinement (DeepSeek-V3).
Instruction Paradigms: The model is trained on three interaction formats:
1. Single-Question Single-Answer: Standard classification/regression.
2. Multi-Question Multi-Answer: Predicting multiple attributes simultaneously.
3. Open-Ended Generation: Free-form description of subject characteristics (demographics, cognitive state, disease).

3. Key Contributions

First LLM-Aligned fMRI Foundation Model: fMRI-LM is the first framework to map resting-state and task-independent brain activity into a token space compatible with pretrained LLMs, enabling a unified interface for modeling and instruction tuning.
Synthetic Descriptive Corpus: The authors constructed a large-scale corpus that translates low-level imaging features (connectivity, gradients, graph metrics) into structured text. This acts as a "bridge" between neural organization and cognitive semantics, providing the necessary language supervision.
Superior Generalization and Efficiency: The model demonstrates strong zero-shot and few-shot capabilities. It achieves state-of-the-art performance across diverse datasets (UKB, HCP, ADNI, ADHD200, ABIDE2) and can be efficiently adapted using LoRA (Low-Rank Adaptation) with minimal parameters.

4. Experimental Results

The model was evaluated on seven datasets covering sex classification, age prediction, fluid intelligence, Alzheimer's disease (AD), ADHD, and Autism.

Performance vs. Baselines: fMRI-LM significantly outperforms supervised baselines (BrainNetCNN, BrainGNN) and recent foundation models (BrainLM, Brain-JEPA, BrainMass) on most tasks.
- Example: On UKB Sex classification, fMRI-LM achieved 94.89% accuracy vs. 88.72% for BrainLM and 84.90% for SWiFT.
- Example: On ADNI AD prediction, it achieved 77.92% accuracy (vs. 78.82% for BrainLM, but with superior generalization across tasks).
Versatility: The model maintains high performance across Multi-Question and Open-Ended paradigms, showing it can handle complex reasoning and free-form generation without significant degradation.
Few-Shot Learning: In zero-shot settings, performance is modest, but with only 2–4 labeled samples, the model adapts rapidly, reaching performance comparable to full fine-tuning on tasks like fluid intelligence prediction.
Parameter Efficiency: Using LoRA for tuning (freezing the base LLM) yields results comparable to full fine-tuning, proving the model's scalability.

5. Significance and Impact

Unified Brain Modeling: fMRI-LM establishes a scalable pathway toward a "universal" model for fMRI, moving away from siloed, task-specific models toward a single framework capable of understanding structural and semantic aspects of brain activity.
Interpretability: By aligning fMRI with language, the model offers a new way to interpret neural data through natural language, potentially making complex brain patterns more accessible to clinicians and researchers.
Cross-Modal Transfer: The success of this approach suggests that the representational structures of LLMs can serve as strong priors for modeling biological signals, opening doors for similar applications in EEG, MEG, and other neuroimaging modalities.
Clinical Potential: The ability to perform multi-task diagnosis (e.g., predicting AD, APOE4 status, and cognitive scores simultaneously) from a single scan with minimal data requirements holds promise for scalable clinical screening tools.

In conclusion, fMRI-LM represents a paradigm shift in neuroimaging analysis, leveraging the reasoning power of LLMs and a novel synthetic text-corpus strategy to create a foundational model that bridges the gap between raw neural signals and high-level cognitive semantics.