🧬 biology

Beyond Prediction Accuracy: Target-Space Recovery Profiles for Evaluating Model-Brain Alignment

This paper introduces a unified framework that evaluates model-brain alignment by quantifying which specific reproducible dimensions of brain response space are recovered, revealing that prediction accuracy alone can mask significant mismatches between artificial vision models and the human visual cortex.

Original authors: Ken Nakamura, Tomoya Nakai, Ryuto Yashiro, Ayumu Yamashita, Kaoru Amano

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ken Nakamura, Tomoya Nakai, Ryuto Yashiro, Ayumu Yamashita, Kaoru Amano

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are trying to teach a robot to see the world the way a human does. Usually, scientists check if the robot is doing a good job by asking a simple question: "How often does the robot guess the right answer?"

If the robot gets 90% of the answers right, we say, "Great job!" But this paper argues that getting the right score isn't enough. It's like a student who gets a perfect grade on a math test by memorizing the answers without actually understanding the formulas. They got the score, but they didn't learn the right things.

Here is a simple breakdown of what this paper does, using some everyday analogies:

1. The Problem: The "Score" is a Black Box

Currently, when we compare a computer vision model (the robot) to a human brain, we just look at one number: Prediction Accuracy.

The Analogy: Imagine two chefs are trying to recreate a complex soup. You taste the soup and say, "Chef A got the flavor 95% right, and Chef B got it 95% right." You declare them tied.
The Reality: But what if Chef A got the flavor right by using too much salt and no pepper, while Chef B used too much pepper and no salt? They both tasted "good" (high score), but they used completely different ingredients to get there. The score hides the fact that they are making the soup in very different ways.

The paper says: "We need to know which ingredients (or brain signals) the model is actually using, not just if the final taste is good."

2. The Solution: The "Reproducible Blueprint"

To fix this, the researchers created a new way to look at the brain's activity. They used a dataset where the same people looked at the same pictures many, many times.

The Analogy: Imagine you are trying to map a city. If you walk through the city once, you might get lost or take a weird shortcut. But if you walk through the city 20 times with different friends, you can figure out which streets are the main, reliable roads that everyone uses, and which are just random, one-off detours.
The Science: The researchers used these repeated walks (fMRI scans) to build a "Reproducible Target Reference." This is a map of the "main roads" of the brain—the parts of the brain's response that are stable, reliable, and happen every time we see a specific image.

3. The New Test: The "Recovery Profile"

Instead of just asking, "How accurate is the robot?", they now ask, "Which parts of the brain's 'main road map' did the robot actually travel?"

The Analogy: Let's go back to the chefs. Instead of just tasting the soup, you now look at their recipe books.
- Chef A (The Robot): You check their recipe. You see they used the main spices (the reliable brain signals) perfectly.
- Chef B (Another Robot): You check their recipe. You see they got the same flavor score, but they used weird, random spices that don't match the main road map at all.
- The Result: Even though they had the same "taste score," you can now see that Chef A is actually a better mimic of the human brain because they used the right ingredients.

The paper calls this a "Recovery Profile." It shows a curve that tells you: "This model recovered the top 10 most important brain signals very well," or "This model only recovered the 1st signal and missed the rest."

4. The Big Discovery: "Smart" vs. "Random"

The researchers tested this on computer models. They compared:

Pre-trained Models: Robots that have already "studied" millions of photos (like a student who read a textbook).
Random Models: Robots that have the same structure but have never seen a photo (like a student who just opened the textbook to a random page).

The Surprise:
Sometimes, the "Random" robot and the "Pre-trained" robot got almost the exact same Prediction Accuracy (the taste score).

Old View: "They are equal."
New View (Using the Recovery Profile): "No! The Pre-trained robot used the 'main roads' of the brain. The Random robot got lucky with the score but used a completely different, weird path that doesn't match how human brains actually work."

5. The Human Reference

To make sure their new test is fair, they also compared the robots to other humans.

The Analogy: Before judging the chefs, you ask, "If a human chef tries to make this soup, what ingredients do they usually use?"
The researchers found that when one human looks at a picture, their brain activity can be predicted by looking at another human's brain activity. This creates a "Human Reference."
Now, they can say: "This robot matches the human reference perfectly," or "This robot matches the human reference poorly, even if its score is high."

Summary

This paper introduces a new tool to stop us from being fooled by high scores.

Before: We only checked How Well a model predicts the brain.
Now: We check How Well AND Which Parts of the brain the model actually understands.

It's like moving from just checking a student's final grade to actually checking their homework to see if they truly understood the lesson or just guessed the right answers. The paper shows that two models can have the same grade but be learning (or failing) in completely different ways.

Technical Summary: Beyond Prediction Accuracy: Target-Space Recovery Profiles for Evaluating Model–Brain Alignment

Problem Statement

Current evaluations of artificial vision models against the human visual cortex rely heavily on prediction accuracy (e.g., correlation between model representations and fMRI responses). While useful for ranking models, a single scalar accuracy score fails to reveal which dimensions of the target brain's response space are actually recovered by the prediction. A model might achieve high accuracy by capturing only a subset of response directions or by matching directions that differ from those recovered by other models with similar accuracy. Furthermore, measured brain responses contain both reproducible signals and noise; without distinguishing between them, it is difficult to determine if a model is aligning with the true, stable structure of the brain or merely fitting noise.

Methodology

The authors introduce a target-space framework that shifts the evaluation focus from scalar scores to the structural overlap between predicted responses and a reproducible target reference.

1. Reproducible Target Reference

Using repeated fMRI measurements (multiple trials of the same images), the authors define a reference that captures dimensions of the target brain's response space that are consistently recoverable across independent trial splits.

Construction: Repeated trials for held-out images are split into two averaged views. Target-to-target predictions are fit in both directions to derive target-side bases. The reproducible target reference is the average of the projectors derived from these splits.
Properties: Unlike PCA, which ranks dimensions by variance, this reference ranks dimensions by predictive reproducibility. Its eigendecomposition yields target-reference directions ( $u_j$ ) and weights ( $\omega_j$ ), where larger weights indicate directions more consistently recovered across trial partitions.

2. Predictive Subspace and Recovery Profiles

For any source (another subject's brain or a vision model), the framework evaluates the source-induced predictive subspace in the target response space.

Source Induction: A ridge-regularized low-rank linear fit maps the source to the target, inducing an orthonormal target basis ( $Q_s$ ) and a projector ( $P_s$ ).
Recovery Profile: The alignment is quantified by how much the predictive subspace overlaps with the reproducible target reference.
- Directional Coverage: Measures if a specific target-reference direction $u_j$ lies within the predictive subspace.
- Top- $k$ Reference Coverage: A weighted average of the coverage of the top $k$ target-reference directions. This curve summarizes which reproducible dimensions are recovered.
- Profile Mean: A scalar summary of the recovery curve's height.

3. Human Reference via Brain-to-Brain Prediction

To establish a baseline for "brain-like" recovery, the authors use brain-to-brain prediction. Responses from non-target subjects are used to predict the target subject's responses. The resulting recovery profile serves as a human reference, indicating which target-reference dimensions are typically recoverable from other human brains. Models are evaluated relative to this human reference.

4. Experimental Setup

Dataset: A subset of the Natural Scenes Dataset (NSD) involving 8 subjects viewing 515 natural images repeatedly.
Targets: Visual cortex ROIs (V1v, V1d, V2v, V2d, V3v, V3d, hV4) in both hemispheres.
Sources:
- Brain: Responses from other subjects.
- Models: ResNet-18/50, VGG-16, and ViT-B/16 in both pretrained (ImageNet) and randomly initialized states.
Protocol: Strict separation of training and evaluation. Outer-test images define the reference and evaluate accuracy; they are never used for model selection or hyperparameter tuning.

Key Results

1. Low-Dimensional Reproducible Structure

Repeated target-brain responses define a low-dimensional reproducible target reference. The reference weights are concentrated in a small number of dimensions (the first three account for ~88.6% of normalized weight), with a median effective rank of 5.12. This structure reflects predictive reproducibility rather than just response variance.

2. Prediction Accuracy Masks Structural Differences

The study demonstrates that prediction accuracy alone can mask model–brain mismatches.

Case Study: Pretrained and randomly initialized models with nearly identical prediction accuracy ( $|\Delta \text{accuracy}| \le 0.01$ ) showed distinct recovery profiles.
Quantitative Gap: Across 29 same-architecture pretrained–random pairs, 79% exceeded the human-variation threshold for profile-mean differences, favoring the pretrained models. This indicates that similar accuracy scores can correspond to the recovery of fundamentally different response dimensions.

3. Pretraining Broadens Recovery

ImageNet pretraining significantly alters the recovery profile. Pretrained models recover a broader set of target-reference dimensions compared to their randomly initialized counterparts.

Profile Mean Difference: Pretrained models exceeded random controls by a mean of 0.177 in profile mean (95% CI: 0.162–0.192), a gap much larger than the variability introduced by random initialization seeds.
Dimensional Specificity: This gain is not limited to the leading dimension; pretrained models show higher coverage across the top 10 dimensions.

4. Brain-to-Brain as a Diagnostic Reference

Brain-to-brain comparisons provide a structured human reference that is more informative than a scalar "noise ceiling." While brain-to-brain prediction accuracy correlates with the overall height of the recovery curve, the shape of the curve (which dimensions are recovered) varies and provides a specific diagnostic for how well a model aligns with human neural structure.

Significance and Claims

The paper claims that prediction accuracy is insufficient for diagnosing model–brain alignment because it does not specify which reproducible brain response dimensions are recovered.

Diagnostic Power: The proposed framework makes explicit which reproducible dimensions are recovered, revealing mismatches that scalar scores hide (e.g., between pretrained and random models with equal accuracy).
Human-Centric Evaluation: By using brain-to-brain prediction to define a human reference, the framework moves beyond abstract benchmarks to a diagnostic comparison of what is recovered from the target brain.
Conditional Interpretation: The authors emphasize that high recovery of the reproducible target reference is conditional evidence of alignment. It indicates that a source-to-target fit recovers reproducible dimensions for a specific ROI, dataset, and readout class, but does not claim the entire model representation is "brain-like."

The work contributes a unified framework for evaluating both model–brain and brain–brain alignment, arguing that understanding the structure of recovery is as critical as the magnitude of prediction accuracy in NeuroAI.