🧬 biology

Quantifying task-relevant representational similarity using decision variable correlation

This paper introduces Decision Variable Correlation (DVC) to quantify task-relevant representational similarity, revealing that deep neural networks trained on image classification exhibit lower alignment with monkey V4/IT activity than with other models, and that increasing model performance or using adversarial training fails to bridge this gap in task-relevant dimensions.

Original authors: Yu Eric Qian, Wilson S. Geisler, Xue-Xin Wei

Published 2026-03-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yu Eric Qian, Wilson S. Geisler, Xue-Xin Wei

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are trying to figure out if a super-smart robot and a human (or a monkey) are "thinking" the same way when they look at a picture.

For years, scientists have been comparing Artificial Intelligence (AI) models to brains. The common belief was: "The better the AI gets at recognizing pictures, the more its brain should look like a human brain."

But this new paper says: "Actually, the opposite might be true."

Here is the story of how they found this out, explained simply.

1. The Old Way: Counting Mistakes (The "Scorecard" Method)

Previously, scientists compared AI and brains by looking at their final answers.

The Analogy: Imagine two students taking a math test.
- Student A (The AI): Gets 99% right.
- Student B (The Brain): Gets 95% right.
- The Old Method: They looked at the answer sheets. If both students got Question 5 wrong, they counted that as a "match." If they got it right, that's a match.
- The Problem: This method is tricky. If Student A is a genius who guesses randomly on hard questions, and Student B is a normal student who guesses randomly, they might accidentally get the same wrong answers. Or, if Student A is too perfect, they might never make the same specific mistakes as the human, making them look less similar, even if they are thinking similarly.

2. The New Way: The "Decision Variable Correlation" (DVC)

The authors of this paper invented a new tool called DVC. Instead of just looking at the final answer (Right/Wrong), they peeked inside the "thinking process" to see how the decision was made.

The Analogy: Imagine two detectives looking at a crime scene photo.
- The Old Method: Did they both arrest the same suspect? (Yes/No).
- The New Method (DVC): We look at their internal "gut feeling" scores for every single photo.
  - Detective A looks at a photo of a cat and feels a 90% certainty it's a cat.
  - Detective B looks at the same photo and feels a 92% certainty it's a cat.
  - High Similarity: If their "gut feelings" rise and fall together for every single picture, they are using the same strategy.
  - Low Similarity: If Detective A feels sure about cats but unsure about dogs, while Detective B feels unsure about cats but sure about dogs, their "gut feelings" are out of sync, even if they both guessed the right answer in the end.

Why is this better? It ignores whether they got the answer right or wrong. It only cares if they are thinking in the same direction for the same reasons.

3. The Big Surprise: The "Smartest" AI is the Most Alien

The researchers tested this new method on:

Monkeys: Looking at pictures of animals, cars, and faces.
AI Models: Hundreds of different computer vision models trained on a massive database called ImageNet.

The Results:

Monkey vs. Monkey: When two monkeys looked at the same picture, their internal "gut feelings" were very similar. They were on the same wavelength.
AI vs. AI: Different AIs were somewhat similar to each other, but not perfectly.
AI vs. Monkey: This was the shocker. The more accurate the AI was at the task, the less similar its thinking was to the monkey.

The Metaphor:
Imagine a video game.

The Monkey plays the game using "human-like" intuition. It sees a car and thinks, "That's a car because of the wheels and the shape."
The AI plays the game using "alien" logic. It sees a car and thinks, "That's a car because of the specific pattern of pixels in the background."
The Twist: As the AI gets better and better at winning the game (getting higher scores), it stops using "human-like" logic and starts using even more "alien" logic. It finds shortcuts that work perfectly for the computer but make no sense to a biological brain.

4. Did "Training Harder" Help?

The scientists tried to fix this gap by:

Adversarial Training: Teaching the AI to be robust against tricks (like making it harder to fool).
Bigger Datasets: Feeding the AI more pictures (like ImageNet-21k instead of ImageNet-1k).

The Result: It didn't help. In fact, making the AI "smarter" or "more robust" actually made it less like the monkey's brain. The AI and the brain seem to be taking two completely different paths to solve the same problem.

5. Why Does This Matter?

This paper suggests that we might have been looking at AI and brains the wrong way.

The Old Hope: "If we just make AI bigger and smarter, it will eventually become like a human brain."
The New Reality: "Making AI bigger and smarter might actually make it more alien."

The Takeaway:
To build AI that truly understands the world like a human (or a monkey), we might need to stop just chasing high scores on tests. Instead, we need to design AI that learns using the same "noisy," imperfect, and biological rules that our brains use. We need to stop building "perfect" machines and start building "biological-like" machines.

In short: The paper introduces a new way to measure "thinking style" rather than just "test scores," and it reveals that the smarter our computers get, the more they seem to think like aliens rather than like us.

1. Problem Statement

The field of computational neuroscience seeks to understand how well Deep Neural Networks (DNNs) model the primate visual system. Previous studies have employed various metrics to compare DNN representations with neural data (e.g., Representational Similarity Analysis - RSA, linear regression) or behavioral data (e.g., Cohen's Kappa, error consistency).

However, these existing methods face significant limitations:

Task-Irrelevant Noise: Methods like RSA measure the geometric structure of representations but are blind to whether those features are actually used to solve the specific task. They can be confounded by task-irrelevant dimensions.
Confounding Factors: Behavioral metrics like Cohen's Kappa conflate decision consistency with accuracy differences and decision biases. High accuracy differences between a model and a human/monkey can artificially lower Kappa scores, while shared biases can artificially inflate them.
Contradictory Findings: Literature contains conflicting trends. Some studies suggest DNNs converge to brain-like representations as they scale, while others suggest a divergence or a plateau. The lack of a principled, task-relevant metric makes it difficult to resolve these contradictions.

The authors propose that the core issue is the need for a metric that isolates task-relevant representational similarity—specifically, the consistency of the decision strategies used by two observers (brains or models) on a per-trial basis, independent of their overall accuracy or decision biases.

2. Methodology: Decision Variable Correlation (DVC)

The authors introduce Decision Variable Correlation (DVC), a method derived from Signal Detection Theory (SDT) and generalized for high-dimensional neural representations.

Core Concept

In SDT, an observer solves a binary classification task by projecting sensory input onto a Decision Variable (DV) axis and comparing it to a criterion. DVC measures the correlation between the DVs of two observers across trials.

High DVC: The two observers use similar encoding and decoding strategies to make decisions, even if their absolute accuracy differs.
Low DVC: The observers rely on different features or strategies.

Implementation Steps

Optimal Linear Decoding: For a given pair of image classes (e.g., cat vs. dog), the authors use Linear Discriminant Analysis (LDA) to find the optimal decision axis that maximizes class separation in the high-dimensional representation space (e.g., V4/IT neural spikes or DNN penultimate layer activations).
Projection: Each image's representation is projected onto this axis to generate a continuous Decision Variable (DV) for that trial.
Correlation Calculation: The Pearson correlation is computed between the DVs of Observer A and Observer B across all trials.
Handling High Dimensions & Noise:
- Dimensionality Reduction: To prevent LDA instability in high-dimensional spaces with few samples, representations are first reduced via PCA (to a fixed number of components, e.g., 25) before LDA.
- Split-Half Noise Correction: Measurement noise attenuates correlation estimates. The authors employ a split-half procedure where data is divided into two independent sets. They calculate the cross-observer correlation and normalize it by the geometric mean of the within-observer (split-half) reliabilities. This yields an unbiased estimate of the true underlying correlation ( $\rho_{corrected}$ ).
Aggregation: DVC is calculated for all pairs of classes in a multi-class task, and the final score is the average of these pairwise correlations.

3. Key Contributions

A New Metric: Development of DVC as a principled, task-relevant measure of representational similarity that is invariant to decision biases and accuracy differences.
Generalization of SDT: Extending the concept of decision variable correlation from binary choice behavioral data to high-dimensional internal neural representations.
Resolution of Methodological Conflicts: Demonstrating that previous contradictions in model-brain alignment literature often stem from the choice of metric (e.g., Cohen's Kappa vs. DVC) and the handling of decoder biases.

4. Key Results

The authors evaluated DVC using:

Neural Data: Recordings from macaque V4 and IT areas (passive viewing of naturalistic scenes).
Models: 43 DNNs trained on ImageNet-1k, 9 adversarially trained "robust" models, and 13 models trained on larger/multimodal datasets (ImageNet-21k, JFT-300M, CLIP).

Findings:

Monkey-Monkey Consistency: DVC between different macaques is high (~0.57), indicating consistent decision strategies across biological subjects.
The "Performance Paradox": Contrary to the hypothesis that better-performing models are more brain-like, there is a strong negative correlation ( $\rho = -0.70$ $ρ = - 0.70$ ) between a model's ImageNet-1k Top-1 accuracy and its DVC with monkey IT/V4.
- Interpretation: As models become more accurate on ImageNet, they diverge from the specific decision strategies used by primates.
Model-Model vs. Model-Monkey:
- Models from the same family (e.g., ResNets) have high DVC with each other.
- However, Model-Monkey DVC is consistently lower than Model-Model DVC.
Robustness Training Fails: Adversarially trained models (designed to be robust to perturbations) show lower DVC with monkeys than standard models, despite having higher internal consistency with other robust models.
Data Scale Fails: Models trained on larger datasets (ImageNet-21k, JFT-300M) or multimodal data (CLIP) do not improve DVC with the brain; in fact, they often show lower similarity.
Comparison with Cohen's Kappa:
- When applied to the same data, Cohen's Kappa shows different trends (e.g., suggesting rich-data models are more aligned).
- The authors demonstrate that Kappa is highly sensitive to accuracy differences and decoder biases. When using an optimal linear decoder (logistic regression) instead of the aggregated probability decoder used in prior studies, Kappa values drop and align more closely with DVC trends, revealing that previous high Kappa scores were artifacts of shared biases.

5. Significance and Implications

Divergence of Strategies: The results suggest a fundamental divergence between the task-relevant representations learned by DNNs trained on ImageNet and those evolved in the primate visual system. Simply scaling up data or improving robustness does not bridge this gap.
Critique of Current Benchmarks: The paper challenges the prevailing assumption that "better accuracy = better brain model." It suggests that high-performing models may be exploiting statistical shortcuts (e.g., texture bias) that are irrelevant or suboptimal for biological vision.
Methodological Shift: DVC offers a more rigorous tool for the field. By focusing on how a decision is made (the decision variable) rather than just what the decision is (the label), researchers can better isolate the representational geometry that truly matters for behavior.
Future Directions: The authors suggest that to create better brain-like models, future training should focus on:
- Datasets that better match the statistical distribution of natural evolution (ecological validity).
- Incorporating internal noise and low-level visual properties (optics, foveation) into training procedures.

In summary, the paper argues that current deep learning models, despite their superior classification accuracy, are learning fundamentally different decision strategies than primates. DVC provides the mathematical framework to quantify this divergence, revealing that "brute-force" scaling has not yet led to true biological alignment.