Evidence from fMRI Supports a Two-Phase Abstraction Process in Language Models

This paper demonstrates that fMRI evidence supports a two-phase abstraction process in large language models, revealing that the intermediate layers' superior ability to predict brain responses arises from their inherent compositionality and intrinsic dimensionality, which evolve and compress during training.

The Gist

Imagine you are trying to understand how a human brain works when it listens to a story. Scientists have discovered that if they look at the "middle layers" of a Large Language Model (like the AI behind this chatbot), those layers predict brain activity better than the final output layer.

But why? Why is the middle of the AI's brain the best match for the human brain, rather than the part that actually spits out the next word?

This paper argues that the answer lies in a two-phase process that happens as the AI learns to speak. Here is the story of that discovery, explained simply.

The Two-Phase Journey: Building vs. Guessing

Think of a Large Language Model (LLM) not as a single brain, but as a factory assembly line with many stations.

Phase 1: The "Composition" Phase (The Art Studio)
In the early stations of the factory, the raw materials (words) are being taken apart and reassembled into complex structures. The AI is figuring out grammar, meaning, relationships between ideas, and the "big picture."

• The Metaphor: Imagine an artist in a studio. They are mixing paints, sketching outlines, and building a complex sculpture. They are creating a rich, high-dimensional representation of the world. This is where the "abstract" thinking happens.
• The Finding: The paper shows that the human brain is most similar to the AI during this phase. Our brains are busy building these rich mental models of what we are hearing.

Phase 2: The "Prediction" Phase (The Casino)
As the data moves down the assembly line to the final stations, the goal changes. The AI stops trying to build a complex world and starts focusing on one specific task: guessing the next word.

• The Metaphor: Imagine the artist is now a gambler at a casino. They aren't painting anymore; they are just betting on what card comes next. To win, they need to narrow their focus. They discard the extra details and zero in on the single most likely outcome.
• The Finding: The final layers of the AI are great at guessing the next word, but they are bad at matching the human brain. Why? Because the human brain doesn't just guess the next word; it holds onto the whole story, the emotions, and the context.

The "Sweet Spot" in the Middle

The researchers found a "sweet spot" in the middle of the AI's layers.

• The Peak: There is a specific layer where the AI has finished building its complex understanding (Phase 1) but hasn't yet started narrowing its focus to just guessing the next word (Phase 2).
• The Evidence: At this exact "peak" layer, the AI's internal map looks almost identical to the human brain's map.
• The Shift: As the AI gets trained longer and becomes smarter, this "sweet spot" actually moves earlier in the line. The AI gets so good at building its understanding that it doesn't need as many steps to get there before it starts guessing.

The "Dimensionality" Clue

How did they know this? They used a concept called Intrinsic Dimensionality.

• The Analogy: Imagine a crumpled piece of paper.
  • Low Dimensionality: A flat sheet of paper is simple. It only has length and width.
  • High Dimensionality: A crumpled ball of paper is complex. It has folds, curves, and depth in every direction.
• The Discovery: The "Composition" phase (the Art Studio) is like the crumpled ball—it is full of complex, high-dimensional information. The "Prediction" phase (the Casino) is like flattening the paper back out to make a simple bet.
• The Result: The human brain is also a "crumpled ball" of complex information when listening to stories. Therefore, the AI layer that is also a "crumpled ball" (high dimensionality) matches the brain best. The layer that is "flattened out" (low dimensionality, just guessing) does not match.

Why This Matters

1. It's Not Just About Guessing: For a long time, people thought AI and brains were similar because they both try to predict the next word. This paper says: No. The similarity comes from the building phase, not the guessing phase.
2. A New Way to Read Brains: If we want to build better tools to read human thoughts or understand language processing, we shouldn't just look at the AI's final answer. We should look at the "middle layers" where the complex ideas are being formed.
3. Training Changes Things: As AI models get bigger and smarter, they get better at the "Art Studio" phase faster. This means the "sweet spot" where they match the human brain moves earlier in the process.

The Bottom Line

The human brain and AI are similar not because they are both good at guessing the next word, but because they both go through a magical phase of building a rich, complex understanding of the world before they ever try to speak. The paper proves that this "building phase" is the true secret to why AI and human brains think alike.

Technical Summary

1. Problem Statement

Recent research has established that intermediate hidden states of Large Language Models (LLMs) can predict human brain activity (measured via fMRI) with high accuracy using simple linear transformations. However, the underlying mechanisms driving this similarity remain unclear.

• The Core Question: Why are intermediate layers of LLMs optimal for predicting brain activity, while the final output layers are not?
• The Competing Hypotheses:
  1. Predictive Coding: The similarity arises because LLMs and brains both optimize for next-token prediction (minimizing surprisal).
  2. Abstraction/Composition: The similarity arises because intermediate layers perform a specific type of feature abstraction or composition that mirrors human linguistic processing, distinct from the final prediction task.

The authors aim to disentangle these hypotheses by analyzing the relationship between encoding performance, representational dimensionality, and prediction error across LLM layers.

2. Methodology

The study employs a multi-modal approach combining fMRI data analysis, manifold learning, and LLM interpretability techniques.

A. Data and Models

• fMRI Data: Functional MRI data from 3 human subjects listening to ~20 hours of English podcast stories.
• LLMs:
  • OPT Family: 125M, 1.3B, and 13B parameters.
  • Pythia Family: 6.9B parameters, with 9 checkpoints ranging from 1K to 143K training steps to observe training dynamics.
• Neural Encoding: Linear ridge regression models were trained to map LLM layer activations to voxel-wise brain responses.

B. Key Metrics

1. Encoding Performance: The predictive power of a linear mapping from a specific LLM layer to brain activity.
2. Intrinsic Dimensionality ($I_d$): Measured using GRIDE (Generalized Ratios Intrinsic Dimension Estimator). This captures the complexity of the nonlinear manifold of the representations.
3. Linear Effective Dimensionality ($d$): Measured via PCA (variance cutoff 0.99) and Participation Ratio (PR).
4. Surprisal (Prediction Error): Computed using TunedLens, a method that learns an affine mapping from intermediate layers to the vocabulary space to estimate next-token prediction error without altering the model's weights.

C. Experimental Design

• Cross-Sectional Analysis: Correlating encoding performance and dimensionality across different layers and model sizes.
• Longitudinal Analysis: Tracking how these metrics evolve across training checkpoints (Pythia) to distinguish architectural artifacts from learned behaviors.
• Phase Transition Detection: Using Centered Kernel Alignment (CKA) to measure similarity between layers and identify distinct phases of processing.

3. Key Contributions

1. Identification of a Two-Phase Abstraction Process: The paper provides empirical evidence that LLMs naturally split into two distinct phases:
   • Phase 1 (Composition/Abstraction): Early-to-mid layers where representations become increasingly complex and abstract, peaking in intrinsic dimensionality.
   • Phase 2 (Prediction/Extraction): Later layers where representations compress to facilitate next-token prediction, leading to a drop in dimensionality.
2. Decoupling Prediction from Abstraction: The authors demonstrate that the high correlation between LLM layers and brain activity is driven by the abstraction phase, not the next-token prediction objective. As models become better at prediction, the layers best for prediction and the layers best for encoding brain activity diverge.
3. Training Dynamics: The study shows that the "peak" layer for both encoding performance and intrinsic dimensionality emerges and stabilizes during training, ruling out the possibility that these peaks are merely artifacts of the Transformer architecture (e.g., fixed layer indices).

4. Key Results

• Correlation between Dimensionality and Encoding: There is a strong positive correlation ($\rho \approx 0.85 - 0.96$) between the intrinsic dimensionality ($I_d$) of a layer and its ability to predict brain activity. This holds across different models (OPT, Pythia) and dimensionality metrics.
• The Phase Transition:
  • In OPT-1.3B, encoding performance peaks at Layer 17.
  • At this exact layer, the intrinsic dimensionality ($I_d$) reaches its maximum.
  • Immediately following this peak, next-token prediction loss (surprisal) drops sharply, while encoding performance declines.
  • This indicates a sharp transition from an abstraction phase (high $I_d$, high encoding) to a prediction phase (low $I_d$, low encoding, high prediction accuracy).
• Training Evolution:
  • As Pythia models train, the peak of intrinsic dimensionality and encoding performance shifts and eventually settles at the same layer (Layer 13 in Pythia-6.9B).
  • The dimensionality of all layers generally increases with training, but the relative peak remains consistent.
• Brain Region Specificity: The correlation between $I_d$ and encoding performance is strongest in brain regions associated with higher-level linguistic processing (e.g., temporal and frontal lobes), whereas the primary auditory cortex (low-level processing) does not show this strong correlation.
• Surprisal vs. Encoding: Layers with the lowest surprisal (best at prediction) are often the last layers, which have the lowest encoding performance. This refutes the hypothesis that predictive coding alone drives brain-model similarity.

5. Significance and Implications

• Theoretical Insight: The findings suggest that the "brain-like" quality of LLMs stems from their ability to compose low-level inputs into high-level abstract concepts (the first phase), rather than their ability to predict the next word. This supports the view that human language comprehension relies heavily on compositional abstraction.
• Model Interpretability: The existence of a distinct "abstraction phase" provides a new lens for interpreting LLM internals. It suggests that intermediate layers contain the most semantically rich information, while final layers are specialized for task execution (prediction).
• Future Directions:
  • Improved Encoding Models: The authors propose that combining representations from layers with high intrinsic dimensionality (potentially across the abstraction phase) could yield better brain-encoding models than single-layer approaches.
  • Generalization: While tested on OPT and Pythia, the authors call for validation across more model families to confirm this as a universal property of LLMs.

In summary, the paper argues that the alignment between LLMs and the human brain is not a byproduct of next-token prediction, but rather a consequence of a shared two-phase processing architecture where an initial phase of high-dimensional abstraction mirrors human linguistic cognition.