🧠 neuroscience

Neural prediction decorrelation reveals that adversarial robustness substantially improves DNN prediction accuracy across the entire human auditory cortex

This study introduces neural prediction decorrelation (NPD), a method that synthesizes stimuli to reveal that adversarially robust deep neural networks significantly outperform standard models in predicting human auditory cortex responses across all regions, a superiority that remains undetectable when using only natural sounds.

Original authors: Skrill, D., Feather, J., Norman-Haignere, S. V.

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Skrill, D., Feather, J., Norman-Haignere, S. V.

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine trying to figure out how a car engine works by only listening to it drive down a perfectly straight, empty highway. You might hear the engine hum, the tires roll, and the wind whistle, but you'd miss the complex gears shifting, the fuel mixing, and the brakes engaging because the road is too smooth and predictable. This is a bit like what scientists face when they try to understand the human brain. For decades, researchers have built computer models to guess how our brains process sounds, like speech or music. They test these models by playing natural sounds and seeing if the computer's "guess" matches the brain's actual electrical activity. The problem is that many different computer models, even ones that are built very differently, seem to give the exact same answer when listening to natural sounds. It's like having two different GPS apps that both tell you to turn left on Main Street; you can't tell which one is actually smarter because they're both right for that specific trip.

To solve this, scientists need to find a way to make the models disagree. They need to play a sound that makes one model say, "That's a dog barking," and the other say, "That's a car honking," while the brain is listening. If they can do that, they can see which model is actually closer to how the human brain works. This is the core challenge of sensory neuroscience: how do we tell the difference between two models that look identical when tested on the things we already know? If we can't tell them apart, we can't be sure which one is truly capturing the secrets of the brain, and we can't use the better one to design new sounds or understand hearing disorders.

This paper introduces a clever trick called "Neural Prediction Decorrelation" (NPD) to shake things up. The researchers took two types of deep neural networks (DNNs)—one standard and one that had been trained to be "adversarially robust" (meaning it's harder to trick with weird noise)—and asked them to predict how the human auditory cortex would react to a bunch of sounds. When they used natural sounds, both models performed almost identically, just like the two GPS apps on the straight highway. The models were so similar in their predictions that the brain couldn't tell them apart either.

But then, the scientists used a special algorithm to invent 60 brand-new sounds. These weren't just random noises; they were carefully crafted to force the two models to make completely different predictions. Think of it like a magic trick where the scientist whispers a secret code to the models that makes them argue with each other. When they played these new "NPD sounds" to people in an MRI machine, the results were dramatic. The standard model, which had been great with natural sounds, suddenly got almost everything wrong. Its predictions dropped to near zero. However, the "adversarially robust" model didn't just survive; it kept predicting the brain's reactions with high accuracy, even though it had never heard these weird new sounds before.

The paper suggests that this difference was completely hidden when they only used natural sounds. The robust model had learned a deeper, more flexible way of understanding sound that the standard model missed. It's as if the standard model only memorized the highway, while the robust model actually learned the rules of driving. The researchers also checked to make sure these new sounds weren't just confusing the brain in a weird way. They found that the brain's reaction to these synthetic sounds still fit right into the same "map" as natural sounds, clustering with things like speech or running water. This means the robust model wasn't just lucky; it had found a better way to describe how our brains handle sound, one that works even when the sounds are strange and unpredictable.

The study explicitly argues against the idea that natural sounds are enough to tell us which computer model is best. They show that relying only on natural stimuli can mask huge differences in how models work. They also rule out the idea that the robust model was just overfitting to the training data, because it generalized perfectly to these brand-new, synthesized sounds that it had never seen. The authors are quite sure of their findings, having measured this effect across multiple regions of the auditory cortex in new groups of people who had never heard these sounds before. They suggest that this "robustness" is a key ingredient for building models that can truly control or predict how our brains respond to sound, opening the door to designing sounds that can target specific parts of the brain in the future.

Technical Summary: Neural Prediction Decorrelation Reveals that Adversarial Robustness Substantially Improves DNN Prediction Accuracy

Problem Statement
A central challenge in sensory neuroscience is distinguishing between distinct encoding models that predict neural responses to complex natural stimuli with similar accuracy. While Deep Neural Networks (DNNs) have emerged as powerful tools for modeling neural computations, standard and adversarially robust DNNs often yield indistinguishable predictions for natural sounds, such as speech. This similarity is frequently attributed to shared model structures or limitations in neural measurement resolution. However, this paper posits that the similarity may also arise from the stimulus set itself: if models are tested only on stimuli that elicit correlated predictions, neural data cannot reveal which model better aligns with the brain. The authors hypothesize that adversarial robustness, which improves the alignment between human and model decisions in behavioral tasks, may not be captured by standard natural stimulus sets.

Methodology
To address the limitation of correlated predictions in natural stimuli, the authors developed a method termed Neural Prediction Decorrelation (NPD).

Stimulus Synthesis: Unlike traditional approaches that optimize models for fixed stimuli, NPD optimizes the stimuli themselves. The method synthesizes a new set of sounds designed to decorrelate the predictions of competing encoding models (specifically, a standard DNN and an adversarially robust DNN) across a target neural population (fMRI voxels in the auditory cortex).
Optimization Objective: The algorithm minimizes a loss function that drives the covariance between the predictions of different models toward zero while simultaneously maintaining high variance in the predictions. This ensures the synthesized sounds elicit strong, distinct responses from each model. The target covariance matrix sets off-diagonal elements (covariance) to zero and diagonal elements (variance) to a high value (5× the variance of natural sounds).
Experimental Design:
- Model Training: The authors utilized a standard DNN (CochResNet50) and an identical architecture trained with adversarial robustness (using $l_2$ -norm constraints, $\epsilon=1.0$ ). Both were trained on word recognition in background noise.
- Data Sources: The study leveraged data from three experiments: NH2015 and B2021 (natural sounds) to fit initial encoding models and synthesize NPD sounds, and a new experiment (S2026) to test these sounds on non-overlapping participants.
- Evaluation: The authors measured fMRI responses to 60 synthesized NPD sounds and compared the prediction accuracy of the standard versus robust models. They also analyzed the low-dimensional organization of these responses using UMAP and Principal Component Analysis (PCA).

Key Results

Masking with Natural Sounds: Consistent with prior findings, the standard and robust models showed virtually identical neural prediction accuracy for natural sounds across all tested regions of the auditory cortex (e.g., Heschl's gyrus, planum temporale). The dissimilarity metric between their accuracies was close to zero.
Unmasking with NPD Sounds: When tested on the synthesized NPD sounds, a dramatic divergence emerged. The adversarially robust model maintained strong prediction accuracy (correlations ranging from ~0.54 to 1.0 across regions), whereas the standard model's prediction accuracy fell to near zero. This effect was observed in every tested anatomical region and was robust across held-out participants.
Shared Neural Subspace: Despite the divergent model predictions, the neural response patterns evoked by NPD sounds overlapped significantly with those of natural sounds. Both natural and NPD sounds occupied a shared, interpretable low-dimensional subspace organized by sound categories and acoustic features (e.g., speech-like structures clustered with natural speech).
Generalization: The robust model, fitted only on natural sounds, successfully predicted the neural responses to the synthesized NPD sounds. In contrast, the standard model failed to generalize to this new stimulus space, suggesting its internal representations are limited to the statistics of natural stimuli.

Significance and Claims
The paper claims that NPD provides a general approach for comparing encoding models by synthesizing stimuli that explicitly decouple model predictions. The primary finding is that adversarial robustness substantially expands the predictive power of DNNs beyond natural stimuli.

Model Comparison: The study demonstrates that the apparent equivalence of standard and robust models on natural sounds is an artifact of the stimulus set. NPD reveals that robust models capture neural computations that standard models miss, specifically those required to generalize to non-natural, decorrelated stimulus spaces.
Neural Population Control: The results suggest that adversarial training enables models to support "neural population control"—the ability to synthesize stimuli that evoke specific, targeted neural response patterns. While standard models struggle to control neural activity outside the narrow distribution of natural sounds without heavy regularization, robust models can do so naturally.
Broader Implications: The authors conclude that adversarial robustness is critical for targeting population activity through stimulus synthesis. This expands the utility of DNNs in sensory neuroscience, allowing for the design of experiments that probe neural mechanisms beyond the constraints of naturalistic stimuli.

The paper remains modest in its scope, noting that while NPD reveals a performance gap, it does not necessarily imply that the robust model is the only correct model, but rather that it is superior in its ability to generalize across a broader stimulus space defined by the decorrelation objective.

Technical Summary: Neural Prediction Decorrelation Reveals that Adversarial Robustness Substantially Improves DNN Prediction Accuracy

More like this