📊 statistics

On Surjectivity of Neural Networks: Can you elicit any behavior from your model?

This paper proves that fundamental components of modern neural architectures, such as pre-layer normalization and linear attention, are almost always surjective, implying that widely used generative models like GPT-style transformers and diffusion models can theoretically generate any output, thereby revealing an inherent vulnerability to adversarial attacks and safety risks.

Original authors: Haozhe Jiang, Nika Haghtalab

Published 2026-06-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haozhe Jiang, Nika Haghtalab

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: Can You Make the Machine Say Anything?

Imagine you have a very sophisticated, trained robot that speaks and creates images. You might wonder: "Is there any specific sentence or picture this robot can never produce, no matter what I type or show it?"

In math terms, this is asking if the robot is surjective.

Surjective means: For every possible output (like a specific sentence or image), there is at least one input that will make the robot produce it.
Not Surjective means: There are some "forbidden zones" in the output world that the robot simply cannot reach, no matter how hard you try.

The authors of this paper asked: Do modern AI models have these forbidden zones, or can they technically produce anything?

The Main Discovery: The "Door is Always Open"

The paper finds that for many of the most popular AI architectures today (like the ones powering ChatGPT, image generators, and robot controllers), the answer is: The door is always open.

They prove that these models are almost always surjective. This means that if you pick any output you can imagine—even something harmful, dangerous, or nonsensical—there is mathematically guaranteed to be some input that will make the model generate it.

The Analogy of the Infinite Key Ring:
Think of the AI model as a giant, complex lock. Usually, we think of the "key" as a normal sentence like "Hello, how are you?"
The paper argues that for these modern models, the lock is so designed that every single possible combination of the lock's tumblers (outputs) can be opened by some key (input). Even if that key looks like gibberish, a secret code, or a weirdly formatted image, it exists.

How Did They Prove This? (The "Smooth Slide" Theory)

The researchers didn't just guess; they used a branch of math called Differential Topology.

The Analogy of the Smooth Slide:
Imagine the AI model as a giant, smooth slide.

Older AI models (like those with simple "ReLU" switches) were like slides with sharp corners or dead ends. If you slid down, you might get stuck in a corner and never reach the bottom (certain outputs were unreachable).
Modern AI models (the ones with "Pre-LayerNorm" and "Attention") are like perfectly smooth, curved slides. Because they are so smooth and continuous, the math proves that you can always find a starting point on the slide that leads to any specific spot at the bottom.

The paper specifically highlights two "magic ingredients" in modern AI that make this happen:

Pre-LayerNorm: A technique that normalizes data before processing it. The paper proves that wrapping a function in this makes it impossible to "block off" any part of the output.
Linear Attention: A specific way the model looks at data (used in newer, faster models) that also guarantees you can reach any output.

What This Means for Safety (The "Jailbreak" Reality)

The paper connects this math to a very real-world problem: Jailbreaking.

The Analogy of the "Unbreakable" Fence:
Imagine a safety team builds a fence around a zoo to keep dangerous animals (harmful AI outputs) inside. They train the AI to be polite and refuse bad requests.

The Old View: We thought, "If we train the AI well enough, it will never go over the fence."
The Paper's View: The math says the fence is an illusion. Because the model is surjective, there is a path over the fence for every single animal.

It doesn't mean it's easy to find that path. It might require a very strange, complex, or hidden input (like a specific code or a weird image). But the paper proves that the path exists.

For Text: You could theoretically find a weird prompt that makes a polite AI write a manual on how to build a bomb.
For Images: You could theoretically find a specific pattern of noise that makes an image generator draw a dangerous weapon.
For Robots: You could theoretically find a sequence of sensor inputs that makes a robot arm move in a way that hurts someone.

What the Paper Does Not Say

It is important to stick to what the paper actually claims:

It does NOT say that we can easily find these dangerous inputs. Finding the "key" might be computationally very hard, like finding a needle in a haystack.
It does NOT say that current safety training is useless. Safety training makes the "needle" harder to find, but it doesn't remove the needle from the haystack.
It does NOT say that every single AI model is like this. The paper specifically notes that older models (like simple ones with ReLU activation) or specific types of attention mechanisms do have "dead ends" where certain outputs are impossible. But the modern ones we use today (Transformers, Diffusion models) generally do not.

The Bottom Line

The paper delivers a sobering mathematical truth: We cannot mathematically guarantee that a modern AI will never produce a harmful output.

Because of how these models are built, they are fundamentally capable of generating anything. Safety, therefore, cannot rely on the model's internal "refusal" to be perfect. Instead, we must accept that the potential for harm is built into the very structure of the machine, and we need to manage that risk through other means (like filters and monitoring), rather than hoping the model itself is "safe by design."

Technical Summary: On Surjectivity of Neural Networks

Problem Statement

The paper addresses a fundamental question in the safety and capability of deep generative models: Can any specified output be generated by some input? In mathematical terms, does a trained neural network correspond to a surjective function from its input space to its output space?

While existing safety research focuses on data curation and fine-tuning to constrain model behavior, a critical gap remains: can we guarantee that a trained model will never generate harmful content? If a model is surjective, then for any arbitrary target output (including harmful, dangerous, or copyrighted content), there exists at least one input that produces it. This raises concerns about the inherent vulnerability of modern architectures to "jailbreak" attacks, where adversaries elicit undesirable behaviors. The paper seeks to determine whether surjectivity is a property that holds "almost always" (i.e., for all parameter configurations except a measure-zero subset) in widely used modern neural architectures.

Methodology

The authors adopt a probabilistic perspective combined with differential topology. Rather than asking if all or some parameter settings yield surjectivity, they ask if surjectivity holds for almost all parameter configurations within a fixed architecture. This approach acknowledges that training introduces randomness (via optimizers, data distributions, etc.), making the final model a draw from a high-dimensional distribution.

The core methodological toolkit involves differential topology, specifically:

Brouwer's Fixed Point Theorem: Used to prove the existence of pre-images for continuous functions under specific bounded conditions.
Brouwer Degree Theory: A topological invariant used to prove surjectivity by showing that the degree of a map is non-zero, implying the existence of a pre-image.
Homotopy Invariance: Used to deform complex functions into simpler ones while preserving the degree, allowing for the analysis of complex neural network blocks.

The authors analyze the surjectivity of fundamental building blocks (MLPs, Attention mechanisms, LayerNorm variants) and then extend these results to composite architectures like Transformers and Diffusion models.

Key Contributions and Results

1. Theoretical Guarantees on Architectural Blocks

The paper establishes rigorous conditions under which specific architectural components are almost always surjective:

Pre-LayerNorm is Surjective: Theorem 3.1 proves that any continuous function $f$ wrapped in Pre-LayerNorm (defined as $g(x) = f(\text{LN}(x)) + x$ ) is surjective. This relies on the fact that LayerNorm maps inputs to a bounded set, allowing the application of Brouwer's Fixed Point Theorem. Consequently, MLP and Attention layers wrapped in Pre-LayerNorm are surjective.
Linear Attention (Retention) is Almost Always Surjective: Theorem 3.4 demonstrates that Retention mechanisms (a form of linear attention without softmax) are almost always surjective. The proof reduces the sequence-level problem to a token-level map of the form $f(x) = Mx + (x^\top N x)x$ and uses degree theory to show surjectivity for almost all parameter choices.
Transformers are Almost Always Surjective: Since modern Transformers (e.g., GPT-style) are compositions of blocks wrapped in Pre-LayerNorm, Corollary 4.1 states that the entire Transformer architecture is almost always surjective.

2. Negative Results (Non-Surjectivity)

The paper also identifies architectures that are not almost always surjective, highlighting the importance of specific design choices:

Softmax Attention: Theorem 3.5 proves that standard attention mechanisms (with softmax) are not almost always surjective. Due to the exponential decay of attention weights, certain output pairs (specifically those with bounded first tokens and arbitrarily large second tokens) form a positive-measure unreachable set.
Standard MLPs (ReLU/GeLU): Theorems B.1 and B.2 show that two-layer MLPs with ReLU or GeLU activations are not almost always surjective. Depending on weight configurations, these networks can be restricted to specific subspaces or bounded regions (e.g., outputs bounded below by a constant).

3. Implications for Real-World Models

The authors apply their theoretical findings to concrete generative frameworks:

Language Models: GPT-style Transformers are almost always surjective. This implies that, in principle, any sentence (including copyrighted or harmful text) can be generated by decoding from a suitable input embedding, even if the model was trained autoregressively.
Diffusion Models: Deterministic diffusion models using ODE solvers (where the velocity predictor is a Transformer or U-Net) are almost always surjective from noise space to output space. This suggests inherent vulnerability to adversarial noise vectors that generate harmful images.
Robotics: Policy networks in robotics that use causally masked Transformers are shown to be almost always surjective, implying that any action sequence (including unsafe trajectories) can be induced by some input sequence.

Significance and Claims

The paper claims to provide a formalism for understanding unavoidable vulnerabilities in AI safety.

Foundational Limitation of Safety Training: The authors argue that surjectivity implies that safety training (e.g., RLHF, fine-tuning) cannot theoretically prevent a model from outputting harmful behavior. If a model is surjective, a harmful output exists in the range; the question shifts from whether it can be generated to how efficiently an attacker can find the corresponding input.
Decoupling Risks: The study decouples the risk of eliciting behavior (the domain of jailbreaks) from the risk of the behavior itself (domain-specific dangers). Surjectivity suggests that the ability to elicit any behavior is an existential property of the architecture, independent of the model's specific capabilities or training data.
Caution in Safety Evaluations: The paper warns against drawing conclusions about model safety solely based on output behavior probing. Since surjectivity implies any output is possible in principle, standard evaluations that fail to find harmful outputs may simply be failing to find the specific, potentially complex, inputs required to elicit them.

The authors maintain a modest stance, clarifying that surjectivity is an existential property, not a constructive one. It guarantees the existence of an input for any output but does not guarantee that such an input can be found efficiently or with feasible information. However, they note that the proliferation of successful jailbreaks suggests that computational difficulty is often not a bottleneck in practice.

In summary, the paper posits that the very architectural choices that make modern neural networks powerful and trainable (specifically Pre-LayerNorm and residual connections) also render them fundamentally surjective, creating an inherent theoretical ceiling on the safety guarantees achievable through training alone.