🤖 machine learning

Adversarial Attacks Leverage Interference Between Features in Superposition

This paper proposes that adversarial vulnerability and transferability arise from "superposition" in neural networks, where efficient information encoding forces non-orthogonal representations that cause interference between features, making targeted perturbations inadvertently affect other concepts.

Original authors: Edward Stevinson, Lucas Prieto, Melih Barsbey, Tolga Birdal

Published 2026-06-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Edward Stevinson, Lucas Prieto, Melih Barsbey, Tolga Birdal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Idea: Packing Too Much into Too Small a Box

Imagine you have a very small suitcase (the neural network's internal "brain" or dimensions), but you need to pack a massive wardrobe full of different outfits (the concepts or features the network needs to understand, like "cat," "dog," "truck," or "cloud").

In a perfect world, you would have a huge suitcase with a separate, distinct compartment for every single outfit. But in reality, neural networks often try to be efficient. They try to fit more concepts than they have physical space for. This phenomenon is called Superposition.

To make this work, the network has to squish these concepts together. Instead of having a dedicated "cat" drawer and a "dog" drawer, it has to use the same space for both, just slightly shifted. It's like trying to write two different stories on the same piece of paper by using slightly different angles of ink.

The Problem: The "Crosstalk" Effect

Because these concepts are packed so tightly, they start to interfere with each other. The paper calls this Interference.

Think of it like a crowded radio station. If you tune into a station playing jazz, and the station next to it is playing rock, sometimes you hear a bit of the rock music mixed into the jazz. The signals are "bleeding" into each other because they aren't perfectly separated.

In a neural network, if you try to activate the "cat" feature, the "dog" feature might accidentally get a little bit of energy too, simply because they are sharing the same internal space.

How Hackers Exploit This (Adversarial Attacks)

Usually, this interference is a small price to pay for efficiency. But the paper reveals that hackers (adversaries) can weaponize this crosstalk.

Imagine you want to trick the network into thinking a picture of a Cat is actually a Dog.

Old thinking: You just tweak the pixels of the cat's ears to look more like a dog's.
New finding: The hacker doesn't just look at the "Cat" and "Dog" buttons. They look at the entire crowded suitcase.

Because the "Cat" and "Dog" concepts are squished together, the hacker can push on a "Cat" feature in a way that accidentally pushes on the "Dog" feature harder than expected, while simultaneously suppressing the "Cat" signal. They use the network's own crowded packing against it.

The paper shows that these attacks aren't random noise. They follow a precise mathematical pattern based on how the concepts are packed. If the network packs "Cats" and "Dogs" close together, the attack will look like a specific, predictable distortion.

Why Attacks "Jump" Between Models (Transferability)

You might wonder: "Why does an attack that works on Model A also work on Model B?"

The paper explains this with the Crowded Room Analogy.
Imagine two different people (two different AI models) trying to pack the same set of clothes into two identical small suitcases.

Even if they start with different packing strategies, the laws of physics (the data they are trained on) force them to pack the "shirts" and "pants" in similar ways because those items are related.
Eventually, both suitcases end up with the "Cat" and "Dog" concepts sitting in almost the exact same spot, interfering with each other in the exact same way.

Because the "interference patterns" are the same in both models, a trick that works on one suitcase will almost certainly work on the other. The paper found that when models are trained on similar data, they develop these "packing patterns" so similarly that attacks transfer between them with nearly 100% success.

The "Magic" of the Experiment

The researchers didn't just guess this; they built a tiny, controlled version of this suitcase (a synthetic model) where they could see exactly how the concepts were packed.

They proved that if you force concepts to share space (Superposition), you automatically create vulnerabilities.
They showed that the "perfect" attack they calculated on paper matched exactly what the computer's hacking algorithm (PGD) discovered on its own.
They tested this on real image classifiers (like those that identify cars or animals) and found the same "interference fingerprints" in the attacks.

The Bottom Line

The paper argues that adversarial vulnerability isn't just a bug; it's a side effect of efficiency.

Neural networks are so good at compressing information (fitting a library into a shoebox) that they create "crosstalk" between ideas. Hackers have learned to listen to that crosstalk and use it to break the system. If you want to stop these attacks, you might need to stop packing the concepts so tightly, or learn how to separate them better, rather than just trying to patch the holes in the suitcase.

Technical Summary: Adversarial Vulnerability from Interference Between Features in Superposition

Problem Statement

Despite extensive study, adversarial examples remain poorly understood at a mechanistic level. While existing explanations attribute vulnerability to high-dimensional geometry, non-robust patterns in input data, or decision boundary structures, none provide a representation-level mechanism that explains why specific perturbations succeed or why attacks transfer between models. There is a lack of a predictive account linking robustness interventions to representation structure. This paper addresses this gap by investigating whether adversarial vulnerability stems from efficient information encoding in neural networks, specifically through the phenomenon of superposition.

Methodology

The authors propose a mechanistic framework where neural networks represent more concepts (features) than they have dimensions ( $M \gg d$ ) by encoding them as non-orthogonal directions in activation space. This packing creates interference, where activating one feature affects others. The study validates this hypothesis through three primary settings:

Synthetic Argmax Task: A minimal bottleneck network with a closed-form input-to-class mapping. Here, $k$ classes are compressed into $m$ latent dimensions ( $m < k$ ). The authors control input correlations (uncorrelated, paired, global) to manipulate feature geometry. This setting allows for the computation of ground-truth perturbations and direct comparison between empirical attacks (PGD) and theoretical optima derived from interference geometry.
Realistic Vision Transformers (ViTs): Models trained on CIFAR-10/100, STL-10, and TinyImageNet with engineered dimensional bottlenecks inserted before the classification head. This tests whether the geometric signatures predicted by the theory emerge in deep, nonlinear networks.
Regularized Training without Explicit Bottlenecks: ViT models trained with varying weight decay to induce low-rank representations (effective bottlenecks) without explicit architectural constraints, testing if rank reduction alone triggers the vulnerability mechanism.

Key Analytical Tools:

Linear Representation Hypothesis (LRH): Treating class logits as linear projections of feature vectors.
Interference Metrics: Quantifying the non-orthogonality of class feature vectors ( $w_i^\top w_j$ ) and measuring how perturbations align with the "interference geometry" defined by the difference between target and source class vectors.
Attack Alignment: Comparing the direction of PGD-discovered perturbations against theoretically optimal perturbations derived from the interference geometry ( $\delta \propto W_e^\top (w_k - w_j)$ ).

Key Contributions

Mechanistic Account of Vulnerability: The paper establishes a direct link between superposition geometry and adversarial vulnerability. It demonstrates that data correlations constrain feature arrangements in superposition, which determine interference patterns. These patterns, in turn, dictate which perturbations succeed and why attacks transfer between models.
Superposition as a Sufficient Condition: The authors provide theoretical and empirical evidence that superposition is a sufficient (though not necessary) condition for adversarial vulnerability. Non-orthogonal representations create exploitable interference, complementing existing explanations based on data properties or architectural factors.
Validation in Realistic Settings: The study validates that the geometric signatures predicted by the theory appear in realistic settings (ViTs on standard datasets), demonstrating that classification head interference governs input-space attack structure even in complex, deep networks.

Key Results

1. Exploitation of Interference

In synthetic settings, gradient-based attacks (PGD) systematically exploit interference between superposed features.

Alignment: PGD-discovered perturbations align strongly with theoretically optimal perturbations derived from interference geometry (Cosine Similarity $> 0.92$ across configurations).
Mechanism: Perturbations are not arbitrary; their sign and magnitude are determined by the alignment of input features with the latent attack vector. Features that constructively interfere with the target class are amplified, while those interfering with the source are suppressed.

2. Role of Input Correlations

Input correlations constrain the geometric arrangement of feature vectors:

Uncorrelated Data: Results in highly variable geometries across random seeds.
Correlated Data: Forces features into specific, stable arrangements (e.g., global correlations fix cyclic ordering).
Impact: Stronger correlations reduce geometric degrees of freedom, forcing different initializations toward similar arrangements.

3. Attack Transferability

Attack transferability is governed by shared interference patterns.

Correlation: Transfer rates correlate strongly with geometric similarity.
- Uncorrelated data: ~5% transfer rate.
- Paired correlations: ~33% transfer rate.
- Global correlations: ~98% transfer rate.
Explanation: When models share similar superposition geometry, interference patterns that were constructive in the source model remain constructive in the target model, enabling high transferability.

4. Real-World ViT Validation

Structure Prediction: In ViTs with bottlenecks, the input-space attack profile (how pixels are perturbed) strongly correlates with the latent-space interference profile (how class vectors interfere). Correlation ( $\bar{\rho}$ ) reaches 0.89 for tight bottlenecks ( $m/k=0.2$ ) and drops as the bottleneck relaxes.
Consistency: Interference patterns are consistent across random seeds, reflecting data semantics (e.g., semantically similar classes like 'cat' and 'dog' develop non-orthogonal representations) rather than optimization noise.
Regularization Effects: Even without explicit bottlenecks, increasing weight decay reduces the effective dimension of the network, increasing interference magnitude and attack vulnerability, suggesting standard regularization may inadvertently create adversarial vulnerabilities.

Significance and Claims

The paper claims that adversarial vulnerability can be a byproduct of network representational compression. By packing more features than dimensions, networks introduce interference patterns that adversaries can exploit.

Predictive Power: The framework offers testable predictions: attacks should align with optimal perturbations derived from geometry, and transferability should occur when models share similar interference patterns.
Complementary Explanation: This mechanism does not replace existing explanations (e.g., non-robust features or decision boundary curvature) but complements them by providing a representation-level account of why specific directions are sensitive.
Safety Implications: The findings suggest that representation dimensionality and the degree of superposition are critical components of model safety. Interventions that alter feature geometry (e.g., via regularization or architectural changes) could significantly impact robustness.

The authors remain modest, noting that superposition is sufficient but not necessary for vulnerability, and that the precise mechanisms may vary by attack and model. The work focuses on features with known labels, with future work needed to extend these findings to intermediate layers via feature discovery methods.