🧬 biology

Brain alignment of reasoning and action representations from vision-language and action models during naturalistic gameplay

This study demonstrates that fine-tuning foundation models for action (LAMs) versus reasoning (VLMs) during naturalistic gameplay leads to distinct brain alignment patterns, where LAMs show asymmetric, action-specialized representations in frontal-motor regions while both models outperform reinforcement learning baselines with gains scaling along the cortical processing hierarchy.

Original authors: Subba Reddy Oota, Anant Khandelwal, Khushbu Pahwa, Satya Sai Srinath Namburi, Tanmoy Chakraborty, Bapi S. Raju, Manish Gupta

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Subba Reddy Oota, Anant Khandelwal, Khushbu Pahwa, Satya Sai Srinath Namburi, Tanmoy Chakraborty, Bapi S. Raju, Manish Gupta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are watching a group of people play a classic arcade video game (like Pac-Man or Space Invaders) while inside a giant MRI machine. This machine acts like a super-sensitive camera that takes pictures of their brains every few seconds, showing which parts light up as they see the screen, think about their next move, and press the buttons.

The goal of this paper is to see if modern AI computers "think" about the game in a way that looks similar to how human brains do.

Here is the breakdown of the study using simple analogies:

1. The Players: Three Types of "Minds"

The researchers compared three different types of "minds" trying to understand the game:

The Old-School Robot (RL Baselines): These are traditional AI agents trained only to win by trial and error. They are like a dog learning to press a lever for a treat; they know what to do, but they don't really "understand" the game world or the rules.
The Generalist Observer (VLMs): These are modern AI models (Vision-Language Models) that have read the internet and watched millions of videos. They are like a very smart tourist who can describe the game, explain the rules, and talk about the graphics, but they haven't necessarily been trained to play it perfectly.
The Action Specialist (LAMs): These are similar to the Generalist, but they have been specifically "fine-tuned" or trained on massive amounts of data about how to control computers and play games. They are like a professional gamer who has practiced the controls until they are second nature.

2. The Experiment: The "Prompt" Trick

The researchers didn't just let the AI look at the screen. They gave them specific instructions, or "prompts," to see how that changed their thinking:

The "Action" Prompt: "What is your next move, and why?" (Focuses on doing).
The "Reasoning" Prompt: "What is happening in the game, what are the goals, and what are the threats?" (Focuses on understanding).

They then compared the AI's internal "thoughts" (digital representations) to the human brain scans to see which AI matched the human brain better.

3. The Big Discoveries

Discovery A: The New AI is Much Closer to Humans
The "Old-School Robots" were okay, but the modern AI models (both the Generalist and the Action Specialist) were much better at matching human brain activity.

Analogy: Imagine trying to guess what a human is thinking. The old robot guesses based on simple patterns. The new AI guesses based on a rich understanding of the story, the objects, and the goals. The new AI's "guess" matches the human brain's actual activity much more closely.

Discovery B: The "Thinking" Boost Happens in the "Planning" Parts of the Brain
When the researchers used the "Action" or "Reasoning" prompts, the AI's match with the human brain got even better. But this improvement wasn't equal everywhere.

Analogy: Think of the brain like a city. The "Early Visual Cortex" is the front gate where people first see the game. The "Frontal-Parietal" areas are the city hall where planning and decision-making happen.
The prompts helped the AI match the human brain twice as much in the "City Hall" (planning/decision areas) compared to the "Front Gate" (visual areas). This suggests that when humans play, they aren't just seeing pixels; they are heavily engaged in planning and reasoning, and the AI models capture this best when asked to think about it.

Discovery C: The "Symmetry" vs. "Asymmetry" Surprise
This is the most interesting finding. Even though the Generalist (VLM) and the Action Specialist (LAM) were equally good at predicting the brain activity overall, they were doing it in very different ways.

The Generalist (VLM) is Balanced: When you ask it to "Reason" or to "Act," it uses its brain power roughly equally for both. It's like a Swiss Army knife that is equally good at opening a bottle or cutting a rope.
The Action Specialist (LAM) is Biased: When you ask the Action Specialist to "Reason," it actually struggles to add new value. It seems that its training to "Act" has already absorbed so much information that asking it to "Reason" separately is redundant.
Analogy: Imagine the Action Specialist is a chef who has memorized the entire recipe book. If you ask them, "What ingredients do you need?" (Reasoning), they just repeat what they already know about cooking (Action). The "Reasoning" part of their brain becomes redundant because their "Action" training already covers it. In the human brain, this showed up as the Action Specialist's "Reasoning" signal actually getting in the way of the prediction in the planning areas of the brain.

Discovery D: The "Thinking" Trace is Weaker
The researchers also looked at a newer type of AI that "thinks out loud" (generating a chain of thought before answering). Surprisingly, the part of the AI that generated the "thinking" text was less like the human brain than the part that just gave the final answer.

Analogy: It's like watching someone solve a puzzle. The final move they make matches how a human would move their hand. But the internal monologue they say while solving it ("Hmm, maybe I should try this...") doesn't match the human brain's activity as well as the final decision does.

Summary

The paper shows that modern AI models are becoming much better at simulating how humans think and plan during games. However, simply making an AI better at doing things (Action) changes its internal structure in a specific way: it makes "Reasoning" and "Action" overlap so much that they become indistinguishable in the brain's planning centers. This helps scientists understand that while AI and humans might reach the same "score" (prediction accuracy), the way their internal "brains" are organized can be fundamentally different.

Technical Summary: Brain Alignment of Reasoning and Action Representations in Naturalistic Gameplay

Problem Statement

Understanding how humans and artificial intelligence systems predict and plan through environmental interaction is a central challenge at the intersection of neuroscience and machine learning. While prior brain-encoding studies have successfully aligned artificial models with brain activity during passive language comprehension or visual processing, interactive brain-alignment research has remained largely limited to reinforcement learning (RL) agents and theory-based models. These traditional models are optimized for reward-based action learning and primarily encode action-related features, potentially missing the richer perceptual, semantic, and predictive structures humans use to represent objects, relationships, and future outcomes.

This paper addresses the gap by investigating whether modern foundation models—specifically Vision-Language Models (VLMs) and Large-Action Models (LAMs)—acquire internal "world models" that align with human brain activity during naturalistic, interactive gameplay. The study seeks to determine how action-focused and reasoning-focused prompts shape model representations and whether these representations capture the neural computations underlying human adaptive behavior better than traditional RL baselines.

Methodology

Dataset and Experimental Setup

The study utilizes a naturalistic Atari-style video game dataset containing fMRI recordings from 32 healthy participants playing six different games. The data is preprocessed and projected onto the cortical surface ("fsaverage6"), with analysis focused on 11 Regions of Interest (ROIs) spanning early visual, frontal-parietal, and motor-planning areas.

Models Evaluated

The authors evaluated two families of foundation models alongside two RL baselines:

Vision-Language Models (VLMs): Qwen2.5-VL-7B-Instruct and InternVL3-8B. These models process visual game frames and text prompts to perform reasoning.
Large-Action Models (LAMs): UI-TARS-7B-DPO and OS-Atlas-Pro-7B. These models are fine-tuned from Qwen2-VL on GUI and game-interaction trajectories, specifically optimized for action planning and execution.
Baselines: EMPA (a theory-based RL agent) and D-DQN (a deep Q-network).

Prompting Conditions

To probe different cognitive modes, the models were tested under three conditions:

No-Prompt: Models received only gameplay frames.
Action Prompt: Models were asked to commit to a next action with a brief justification.
Reasoning Prompt: Models were asked to describe game objectives, spatial relationships, and goals without necessarily committing to an immediate action.

Encoding and Analysis

Feature Extraction: Hidden-state representations were extracted from all transformer layers using a sliding window of recent frames to match the temporal resolution of fMRI (TR = 2s).
Voxel-wise Encoding: Bootstrap ridge regression models were trained to predict fMRI responses from model representations. Performance was measured using Pearson Correlation Coefficient (PCC).
Variance Partitioning: A statistical approach was used to decompose the explained variance ( $R^2$ ) into components shared between action and reasoning prompts, and components unique to each prompt type. This allowed the authors to distinguish between models that encode similar information versus those with qualitatively different representational organizations.
Thinking-Mode Analysis: A preliminary study on Qwen3.5 compared the brain alignment of its explicit reasoning traces (Chain-of-Thought) against its final action outputs.

Key Results

1. Foundation Models Outperform RL Baselines

Both VLMs and LAMs significantly outperformed classical RL baselines (EMPA and D-DQN) in voxel-wise brain encoding, even when feature dimensions were matched (e.g., reduced to 8 dimensions).

No-Prompt Performance: VLMs achieved $r \approx 0.264$ and LAMs $r \approx 0.271$ , compared to EMPA ( $r=0.162$ ) and D-DQN ( $r=0.097$ ).
Prompt Effects: Explicit prompting further improved alignment for both model families, with prompted VLMs reaching $r \approx 0.358$ and LAMs $r \approx 0.348$ .

2. Cortical Hierarchy of Prompt-Driven Gains

The improvement in brain alignment driven by prompting was not uniform across the brain.

Frontal-Parietal & Motor Regions: The largest gains were observed in higher-order regions involved in planning and decision-making, specifically the Middle Frontal Gyrus (MFG), Supplementary Motor Area (SMA), Inferior Frontal Gyrus (IFG), and Angular Gyrus (AG).
Visual Cortex: Early visual regions showed gains roughly half the magnitude of frontal regions.
Significance: This hierarchy suggests that prompts primarily enhance representations related to goals, planning, and the "multiple-demand" network rather than just low-level visual processing.

3. Qualitative Dissociation via Variance Partitioning

While VLMs and LAMs achieved statistically equivalent raw prediction accuracy across the whole brain, variance partitioning revealed a fundamental difference in their internal organization:

VLMs (Prompt-Symmetric): The unique variance contributed by action prompts ( $12.5\%$ ) and reasoning prompts ( $13.6\%$ ) was balanced.
LAMs (Action-Asymmetric): Action prompts contributed significantly more unique variance ( $27\%$ ) than reasoning prompts, which contributed negative unique variance ( $-5\%$ ) in the whole brain and frontal-motor cortex.
Interpretation: In LAMs, the action-specialized fine-tuning appears to reorganize representations such that the "reasoning" signal becomes redundant or noise-like in motor-planning regions, as the action supervision already subsumes the necessary information. This dissociation is invisible at the level of raw accuracy but clear in variance decomposition.

4. Thinking-Mode Asymmetry

In the preliminary analysis of Qwen3.5, the model's final action output showed substantially higher brain alignment ( $r=0.355$ ) than its explicit reasoning trace ( $r=0.173$ ). This gap persisted even after controlling for readout position, suggesting that explicit chain-of-thought reasoning traces in current thinking-mode models may not align as closely with human neural dynamics as the model's final action commitment.

Significance and Claims

The paper claims to extend brain-alignment research from passive perception to interactive gameplay, providing the first systematic evaluation of modern VLMs and LAMs in this setting. The primary contributions are:

Superior Representational Richness: Multimodal foundation models encode more brain-relevant information about dynamic game environments than traditional task-optimized RL agents.
Prompt-Conditioned Reorganization: The study demonstrates that prompting can shift model representations to better align with human brain activity, particularly in higher-order cognitive regions.
Action vs. Reasoning Dissociation: The work reveals that action-specialized fine-tuning (LAMs) fundamentally reorganizes multimodal representations toward action-relevant neural computations, creating a prompt-asymmetric structure distinct from general-purpose VLMs. This distinction is only detectable through variance decomposition, not raw accuracy.
World Model Evaluation: By contrasting reasoning-focused and action-focused prompts, the study operationalizes the separation between "understanding the environment" and "committing to actions," showing that this distinction maps onto qualitatively different representational structures in AI models.

The authors conclude that these findings open new directions for studying how foundation models encode the structured, predictive representations that underlie adaptive behavior, while noting limitations regarding the specific model families tested and the preliminary nature of the thinking-mode analysis.