🧬 biology

Task-conditioned probing of instruction-tuned multimodal LLMs: Region-specific brain alignment patterns under naturalistic stimuli

This study demonstrates that instruction-tuned multimodal large language models exhibit significantly stronger and more task-specific alignment with human brain activity during naturalistic movie watching compared to non-instruction-tuned and in-context learning models, suggesting that instruction tuning organizes representations around functional task demands rather than surface semantics.

Original authors: Subba Reddy Oota, Khushbu Pahwa, Prachi Jindal, Satya Sai Srinath Namburi, Maneesh Singh, Tanmoy Chakraborty, Bapi S. Raju, Manish Gupta

Published 2026-05-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Subba Reddy Oota, Khushbu Pahwa, Prachi Jindal, Satya Sai Srinath Namburi, Maneesh Singh, Tanmoy Chakraborty, Bapi S. Raju, Manish Gupta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

The Big Idea: Teaching Computers to "Think" Like Brains

Imagine you have a group of very smart AI assistants (Multimodal Large Language Models, or MLLMs). These AIs can watch movies and listen to audio. The researchers wanted to know: Do these AIs process information the same way human brains do?

To find out, they didn't just ask the AIs questions; they looked at the "brainwaves" (fMRI scans) of real humans watching the same movies. They then tried to predict those brainwaves using the AI's internal thoughts. The goal was to see which AI "brain" matched the human brain best.

The Main Characters: Two Types of AIs

The study compared two types of AI assistants:

The "Raw" AI (In-Context Learning): Think of this as a brilliant student who has read millions of books but has never been given a specific test. If you show them a movie and ask, "What's happening?", they answer based on their general knowledge. They are smart, but they might just be repeating what they've seen before.
The "Trained" AI (Instruction-Tuned): Think of this as the same student, but they have just finished a rigorous training camp where they practiced specific tasks: "Summarize this," "Find the emotion," "Describe the sound." They are now experts at following specific instructions.

The Experiment: The Movie Night

The researchers set up a "movie night" with four human volunteers. While they watched clips from popular movies (like The Wolf of Wall Street), their brains were scanned.

Then, they fed those same movie clips into the AIs.

The Raw AI just looked at the video.
The Trained AI was given 13 different "instructions" for the same video, such as:
- "What are the main events?"
- "Describe the emotions."
- "What is the central conflict?"
- "Identify the sounds."

The researchers then asked: Which AI's internal "thoughts" could best predict what the human brain was doing at that exact moment?

The Surprising Results

Here is what they discovered, using some simple metaphors:

1. The "Trained" AI is a Better Match for the Human Brain
The "Trained" (Instruction-Tuned) AIs were significantly better at predicting human brain activity than the "Raw" AIs.

Analogy: Imagine trying to guess what a friend is thinking. The "Raw" AI is like a friend who just lists random facts about the movie. The "Trained" AI is like a friend who is actively trying to understand the story and the feelings, just like your brain does. The Trained AI's "thoughts" aligned with the human brain about 9% to 20% better than the others.

2. The "Raw" AI is Just Parroting Words; The "Trained" AI is Doing the Work
The researchers found a key difference in how these AIs think:

The Raw AI is very sensitive to the exact words you use. If you say "Describe the video" vs. "Tell me about the video," the Raw AI's internal brain changes a lot because the words are different. It's like a parrot mimicking sounds.
The Trained AI ignores the specific wording and focuses on the job you want done. Whether you say "Summarize" or "Tell the story," its internal brain stays focused on the task.
The Takeaway: Human brains also seem to care more about the task (understanding the story) than the exact words used to ask for it. The Trained AI mimics this human-like behavior better.

3. Different Tasks Light Up Different Parts of the Brain (and the AI)
The study showed that when you give the AI a specific job, it activates different "departments" in its brain, just like humans.

Analogy: Think of the brain as a city with different districts.
- When the AI is asked to detect sounds, its "Audio District" lights up, matching the human auditory cortex.
- When asked to understand a story, its "Language District" lights up, matching the human language areas.
- When asked to recognize objects, its "Visual District" lights up.
The "Raw" AI didn't switch districts as clearly. The "Trained" AI knew exactly which part of the city to use for the specific job.

4. The "Deep" Layers Match the "Deep" Brain
AI models have layers, like a multi-story building.

Bottom floors (Shallow layers): These handle simple things like edges, colors, and basic sounds.
Top floors (Deep layers): These handle complex ideas like stories, emotions, and reasoning.
The Finding: The researchers found that the AI's bottom floors matched the human brain's sensory areas (eyes/ears), and the AI's top floors matched the human brain's thinking areas. This "hierarchy" was much clearer in the Trained AI.

The "Video vs. Audio" Twist

The study also looked at AIs designed specifically for video and AIs designed specifically for audio.

Video AIs: These were the stars of the show. They matched human brain activity very well across the whole brain.
Audio AIs: These did okay, but they mostly matched the human brain's hearing centers. They didn't match the rest of the brain as well as the video models did.
Conclusion: Currently, AI models that can "see" and "hear" together (Video) are much closer to human brain function than models that only "hear" (Audio).

Summary

This paper proves that when we teach AI models to follow specific instructions (like a human teacher would), they stop just "reciting" data and start "thinking" in a way that looks remarkably like how our own brains work. They organize their thoughts by task rather than by wording, and they activate specific brain regions depending on what they are asked to do. This makes them powerful tools for scientists trying to understand how the human brain processes the complex world of movies and stories.

Technical Summary: Task-conditioned probing of instruction-tuned multimodal LLMs

Problem Statement

Recent brain encoding studies have established that multimodal large language models (MLLMs) exhibit higher alignment with human brain activity than unimodal models. Furthermore, instruction-tuned (IT) MLLMs have been shown to generate task-specific representations that align strongly with brain activity. However, prior evaluations have largely been limited to two settings: (1) multimodal models evaluated with unimodal stimuli, or (2) non-instruction-tuned models evaluated with multimodal stimuli.

A critical gap remains in understanding whether instruction-tuning causes IT-MLLMs to organize their representations around functional task demands or if they merely reflect surface semantics. Additionally, there is a lack of systematic probing of IT-MLLMs under naturalistic multimodal stimuli (video with audio) to determine how task-specific, language-guided representations align with brain activity across cortical regions, from early sensory to higher-order cognitive areas.

Methodology

The authors address this gap by estimating brain alignment through the prediction of fMRI responses recorded during naturalistic movie watching (video with audio) using representations from instruction-tuned MLLMs.

Data and Models:

Dataset: The study utilizes the Movie10 dataset (Boyle et al., 2020), a multimodal naturalistic fMRI dataset where four subjects passively watched four different movies. The data is projected onto the "fsaverage6" surface space.
Models Evaluated:
- Instruction-Tuned (IT): Six video MLLMs (e.g., InstructBLIPVideo, Video-LLaVA, Qwen-2.5-VL) and two audio MLLMs (Qwen-2.5-Audio, Kimi-Audio).
- Baselines: Two in-context learning (ICL) video MLLMs, two non-instruction-tuned multimodal models, two unimodal video models, and one unimodal audio model.
Instructions: The models were probed with 13 video-specific task instructions (e.g., action recognition, narrative understanding, commonsense reasoning) and 5 audio-specific instructions (e.g., audio understanding, sound event detection).
Feature Extraction: For IT models, hidden state representations were extracted at each layer based on the input video/audio and the specific task instruction. For ICL and non-IT baselines, zero-shot inference was performed with natural language prompts.

Encoding and Evaluation:

Voxel-wise Encoding: Banded ridge regression models were trained to predict fMRI brain activity from the model representations. The models accounted for hemodynamic response delays using a finite impulse response (FIR) filter.
Metrics: Performance was measured using Pearson Correlation (PC) between predicted and actual brain activity. To standardize performance, results were divided by the estimated cross-subject prediction accuracy, yielding "normalized brain alignment."
Analysis: The study employed Representational Similarity Analysis (RSA) to compare instruction semantic similarity with model representation similarity. It also utilized clustering metrics (Adjusted Rand Index) to assess task-conditioned organization and variance partitioning to analyze shared and unique variance across tasks.

Key Contributions and Results

1. Decoupling of Surface Semantics from Task Structure
The study finds a fundamental difference in how IT and ICL models organize representations:

ICL Models: Show strong coupling (Pearson $r \approx 0.78$ ) between internal representations and surface instruction semantics, indicating shallow text-proximal matching.
IT Models: Exhibit weak coupling to surface semantics ( $r \approx 0.14$ ) but show stronger task-conditioned clustering. This suggests IT models organize representations based on functional task demands rather than the specific wording of the prompt.

2. Superior Brain Alignment of IT Video MLLMs

Overall Performance: IT video MLLMs demonstrate significantly higher brain alignment than ICL models (approx. 9% improvement), non-instruction-tuned multimodal models (approx. 15% improvement), and unimodal baselines (approx. 20% improvement).
Modality Asymmetry: While IT video models show strong alignment across whole-brain, language, and visual regions, IT audio models show gains primarily in the auditory cortex and the middle frontal gyrus (MFG). Audio IT-MLLMs perform comparably to non-instruction-tuned multimodal baselines, indicating room for improvement in capturing brain-relevant semantic information for audio.

3. Region-Specific Task Differentiation
Task-specific instructions yield distinct representations that align with specific functional brain regions:

Video Tasks: "Narrative understanding" and "Visual QA" show higher alignment in language-related regions (angular gyrus, posterior temporal lobe), while "Video understanding" dominates higher-order frontal regions.
Audio Tasks: "Audio captioning" and "sound event detection" align with the auditory cortex, whereas "audio understanding" activates higher-level regions like the inferior frontal gyrus (IFG).
Variance Partitioning: Tasks with high visual load contribute uniquely to visual cortex variance, while high-level semantic tasks contribute uniquely to language network variance.

4. Hierarchical Alignment with Cortical Processing
IT-MLLMs exhibit a layer-wise alignment pattern consistent with the brain's processing hierarchy:

Early Layers: Align with early sensory areas (visual and auditory cortex).
Middle/Deep Layers: Align with higher-order visual areas (LOC, PPA) and language regions (angular gyrus, IFG).
Control Experiments: This hierarchical gradient disappears when using non-natural language prompts (random tokens), confirming that the alignment is driven by instruction semantics rather than generic model structure. Controlled comparisons within model families (e.g., Qwen-2.5-VL IT vs. Pretrained) confirm that instruction tuning consistently improves alignment across all layers.

Significance and Claims

The paper claims to be the first study to use instruction-tuned video and audio MLLMs to model fMRI responses across diverse multimodal tasks. The findings suggest that:

Instruction-tuning transforms MLLMs into probes that organize representations around task demands rather than surface text, leading to stronger alignment with the brain's functional organization.
Task-specific instructions can be used as a controlled lever to vary task goals while holding the stimulus fixed, allowing for the mapping of joint information processing in both artificial and biological systems.
The observed layer-dependent alignment patterns support the use of IT-MLLMs for studying functional specialization under naturalistic stimuli and connecting cognitive modeling with neuroscience.

The authors maintain a modest stance regarding cross-family comparisons, noting that differences in scale, training data, and architecture exist. However, they emphasize that controlled within-family comparisons (IT vs. Pretrained versions of the same model) consistently show positive gains in brain alignment after instruction tuning, isolating the effect of the tuning process itself.