🤖 machine learning

Why Does Robustness Reduce Superposition?

This paper empirically explains that adversarial training reduces superposition by forcing models to abandon non-robust features, thereby decreasing the total number of features that need to be represented within the network.

Original authors: Adam Elimadi

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Adam Elimadi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of artificial intelligence, computer models are often treated as black boxes: we feed them data, they give us an answer, but the inner workings remain a mystery. For years, researchers have been puzzled by a strange vulnerability in these systems. Even the most accurate models can be tricked by "adversarial examples"—tiny, almost invisible changes to an image or sound that cause the computer to make a complete error. For instance, a photo of a panda might be altered by a few pixels in a way the human eye cannot detect, yet the computer suddenly becomes convinced it is looking at a gibbon. Scientists have long known that these models rely on certain patterns in the data to make decisions, but they did not fully understand how those patterns are stored inside the machine's brain, or why the machine would be so easily fooled by them.

Recent work has introduced a concept called "superposition" to explain this storage problem. Imagine a model as a room with a fixed number of shelves. If the model needs to remember more things than there are shelves, it must start stacking items on top of each other, or packing them into the same space. This is superposition: the model is trying to hold more features than it has physical room for. A separate line of research has shown that when models are trained to resist these tricky adversarial examples, they become more robust, but they also seem to stop packing so many things together. They appear to stop superposing. However, no one could explain the mechanism behind this change. Why would teaching a model to ignore a tiny trick cause it to simply stop remembering certain things?

A new study by independent researcher Adam Elimadi, presented at a workshop on AI interpretability, provides a clear answer to this question. The researcher set out to trace the chain of events that happens when a model is trained to be robust. Using simplified computer models that mimic the behavior of larger, real-world systems, the study demonstrates that adversarial training forces the model to abandon specific types of information. The model learns that certain patterns in the data are too fragile to be useful when under attack. These patterns, known as "non-robust features," are like whispers that are easily drowned out by noise. When the model is trained to withstand noise, it realizes that holding onto these fragile whispers is a liability. Consequently, it drops them entirely. Because the model no longer needs to store these discarded features, it has fewer things to pack into its limited space, and the need for superposition naturally disappears.

To prove this, the researcher first observed that models trained to resist attacks consistently represented fewer features than models trained on normal data. In these experiments, the models were given a specific amount of "sparsity," a setting that controls how many features they are allowed to activate. Across a wide range of these settings, the robust models always chose to represent fewer features. The study then looked at the geometry of these features—essentially how they sit in the model's internal space. It found that the features the robust models chose to drop were the ones that caused the most interference with each other. They were the features that, when packed together, created confusion and errors. The models that survived the training kept only the features that fit together cleanly, often aligning them in opposite directions to minimize conflict, while discarding the rest.

The most critical part of the study involved a synthetic dataset where the researcher knew exactly which features were "robust" and which were "non-robust" before the training even began. In this controlled environment, the robust features were strong and stable signals, while the non-robust features were weak and easily disrupted. When the model was trained to resist attacks, it dropped exactly the set of non-robust features every single time, regardless of how much space it had available. Even when the model had plenty of room to store everything, the adversarial training convinced it that the non-robust features were too dangerous to keep. The study confirms that the reduction in superposition is not a random side effect, but a direct result of the model pruning away the specific, fragile features that make it vulnerable to being tricked.

This discovery offers a new way to understand the relationship between security and efficiency in artificial intelligence. It suggests that making a model robust is not just about adding more defenses, but about fundamentally changing what the model considers worth remembering. By forcing the model to confront the fragility of its own knowledge, adversarial training simplifies its internal structure. The model stops trying to hold onto every possible pattern and focuses only on the ones that are stable and reliable. While these findings were established using simplified toy models, they provide a clear, mechanistic explanation for a phenomenon that was previously a mystery. The next step for researchers will be to see if this same process of dropping fragile features and reducing superposition happens in the complex, real-world models that power modern technology.

Technical Summary: Why Does Robustness Reduce Superposition?

Problem Statement
The existence of adversarial examples—imperceptible perturbations that fool accurate models—remains an open research problem. While Ilyas et al. (2019) established that models rely on "non-robust features" (patterns predictive on clean data but fragile under perturbation), this account did not explain the internal representation of these features. Recent work by Gorton & Lewis (2025) connected adversarial vulnerability to superposition (the packing of more features into a representation than the dimensionality allows) and empirically demonstrated that adversarial training reduces superposition. However, Gorton & Lewis (2025) did not provide a mechanistic explanation for why training against perturbations causes a model to reduce the number of features it chooses to represent. This paper aims to supply that missing mechanistic account.

Methodology
The author employs a simplified toy model setup (Elhage et al., 2022; Gorton & Lewis, 2025) consisting of a linear autoencoder with $n=20$ hidden dimensions and $m=100$ input features. The model is trained to reconstruct sparse inputs $x$ under a mean squared error (MSE) loss.

Training Regimes: The study compares two training protocols:
- Standard Training: Minimizing loss on clean data.
- Adversarial Training: Minimizing a weighted loss $L_{adv} = \alpha L(x) + (1-\alpha) L(x_{adv})$ on a mixture of clean and adversarial examples, where $x_{adv}$ is generated via a one-step $\ell_2$ gradient attack.
Sparsity Sweep: Experiments are conducted across a range of sparsity thresholds ( $S \in \{0.78, 0.80, 0.88, 0.90\}$ ) to isolate the effects of feature density on superposition.
Metrics:
- Superposition ( $\Phi$ ): Measured as the squared Frobenius norm of weights per dimension ( $\|W\|_F^2 / n$ ).
- Representational Power ( $P$ ): The total squared Frobenius norm ( $\|W\|_F^2$ ).
- Feature Dropping: Features are classified as "dropped" if their weight column norm $\|W_{:,i}\|_2$ falls below a threshold $\tau = 0.01$ .
- Interference: Calculated via the off-diagonal elements of the Gram matrix $G = WW^\top$ , measuring the degree of non-orthogonality between feature directions.
Synthetic Ground Truth: To explicitly test the relationship between dropped features and robustness, the author constructs a synthetic dataset where features are pre-labeled as "robust" (high amplitude, $a_r=6.0$ ) or "non-robust" (low amplitude, $a_{nr}=0.2$ ) based on the taxonomy of Ilyas et al. (2019) and Li & Li (2025).

Key Contributions and Results

Adversarial Training Reduces Feature Count:
The study confirms that adversarially trained models consistently exhibit lower superposition and representational power than standardly trained models across the sparsity sweep. This reduction is driven by the fact that adversarially trained models drop significantly more features than their standard counterparts.
Geometric Cost of Dropped Features:
The features dropped by adversarially trained models are not random; they are those with the most "costly" interference geometry.
- Dropped features exhibit higher mean total interference (or less negative interference) compared to retained features.
- Retained features cluster near an interference value of $\approx -1$ , corresponding to antipodal directions which minimize cross-feature activation and spurious reconstruction.
- This suggests adversarial training forces the model to prune features that induce harmful interference, favoring a configuration that maximizes orthogonality (or anti-orthogonality).
Direct Link to Non-Robust Features:
Using the synthetic dataset with known ground-truth partitions, the author demonstrates a precise correspondence: the features dropped by adversarially trained models are exactly the set of non-robust features defined in the data generation process.
- Even at sparsity levels where standard models retain all features, adversarially trained models consistently drop the entire set of non-robust features ( $|N|=70$ ).
- This behavior is invariant to sparsity levels, suggesting the model explicitly targets non-robust features to minimize the adversarial objective.

Mechanistic Explanation
The paper proposes a causal chain explaining the reduction in superposition:

Adversarial Training Abandons Non-Robust Features: Non-robust features are highly sensitive to perturbations; their values flip under attack, causing a direct inflation of the reconstruction loss ( $L_{adv}$ ).
Fewer Total Features to Represent: To minimize the adversarial loss, the model learns to discard these high-cost, non-robust features entirely.
Less Superposition: With fewer features requiring representation in the same fixed-dimensional space, the density of features per dimension decreases, resulting in reduced superposition.

Significance and Limitations
The paper claims to provide the first empirical and mechanistic explanation for why adversarial training reduces superposition, bridging the gap between the feature taxonomy of Ilyas et al. (2019) and the representational geometry observed by Gorton & Lewis (2025). The findings suggest that robustness is achieved not just by adjusting weights, but by fundamentally pruning the feature set to eliminate non-robust, interference-prone components.

The author notes that these results are constrained to a toy model setting optimizing reconstruction loss. They identify the validation of this theory on real-world models using mechanistic interpretability tools (e.g., Sparse Autoencoders) as the necessary next step. The paper does not claim to solve the broader problem of adversarial robustness but rather clarifies the internal representational consequences of robust training.

Technical Summary: Why Does Robustness Reduce Superposition?

More like this