🤖 machine learning

The Symmetries of Three-Layer ReLU Networks

This paper establishes a complete framework for characterizing parameter symmetries in three-layer ReLU networks, providing explicit semi-algebraic descriptions of functional equivalence fibers and a polynomial-time algorithm for their decision, while also analyzing how these symmetries induce local conservation laws along gradient flow.

Original authors: Johanna Marie Gegenfurtner, Moritz Grillo, Guido Montúfar

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Johanna Marie Gegenfurtner, Moritz Grillo, Guido Montúfar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a neural network as a complex machine built from layers of switches and levers. In a ReLU network (a very common type of AI), these switches turn "on" if a signal is positive and stay "off" (zero) if it's negative.

The paper you provided asks a fundamental question: If two different machines produce the exact same output for every possible input, are they actually the same machine underneath?

The answer is no. Just like two different recipes can make the exact same cake, two different sets of numbers (parameters) inside a neural network can produce the exact same function. The collection of all these "different but identical" settings is called a fiber.

Here is a breakdown of what the authors discovered, using simple analogies:

1. The "Hidden" Symmetries

In simple, shallow networks, we already knew about two ways to change the numbers without changing the result:

Swapping Neurons: Imagine a row of lightbulbs. If you swap bulb A and bulb B, the room looks the same.
Rescaling: If you turn bulb A up by 2x and the wire connecting it to the next layer down by 2x, the final brightness remains unchanged.

The authors discovered that in three-layer networks, there are new, stranger symmetries that don't exist in simpler networks. These happen because of how layers interact with each other.

2. The "Hiding" Mechanism

The most exciting discovery is about hiding.

Imagine the first layer of the network draws a map of the world with several lines (hyperplanes). The second layer is supposed to look at this map and make decisions.

The Normal Case: The second layer sees all the lines clearly. If you move a line, the second layer notices, and the final output changes.
The Hiding Case: Sometimes, the second layer is arranged in a specific way (like a specific pattern of weights) that acts as a blindfold. It completely ignores a specific line drawn by the first layer.

The Analogy: Imagine you are drawing on a piece of paper (Layer 1). A second person (Layer 2) is looking at your drawing through a specific pair of sunglasses.

If the sunglasses are clear, they see every line you draw.
If the sunglasses are "hiding" a specific line, they can't see it.
The Symmetry: Because they can't see that specific line, you can slide that line left, right, up, or down, and the second person won't notice. You can also flip the line's direction, and they still won't notice. The final result (the "cake") tastes exactly the same, even though the "recipe" (the parameters) has changed.

3. The "Bottleneck" Discovery

The authors focused on a specific type of network called a bottleneck architecture.

The Analogy: Imagine a funnel. The input is wide, the middle layer is narrow (the neck of the funnel), and the output is wide again.
In this specific "funnel" shape, the authors proved they have found every single possible way to change the numbers without changing the function. They created a complete "map" of all the hidden symmetries.

4. What This Means for "Identifiability"

In science, "identifiability" means: "If I see the output, can I figure out the exact settings used to create it?"

The paper shows that for these three-layer bottleneck networks, the answer is usually no, unless the settings are very specific.
They found a simple rule based on signs (positive or negative numbers) in the second layer. If the signs follow a certain pattern, the network is "hiding" information, and you can't tell the difference between the original settings and the "hiding" settings.
Good News: Because they understand these patterns so well, they built a fast computer algorithm (polynomial time) that can look at two sets of numbers and instantly say, "Yes, these produce the same function" or "No, they don't."

5. The "Global" vs. "Local" Surprise

One of the most counter-intuitive findings is about localizability.

The Expectation: You might think, "If Layer 1 is unique, and Layer 2 is unique, then the whole machine must be unique."
The Reality: The authors proved this is false. You can have a network where Layer 1 is unique, and Layer 2 is unique, but when you combine them, they create a new, hidden symmetry that makes the whole thing non-unique.
The Metaphor: It's like two people who are both excellent, unique dancers. But when they dance together, they accidentally create a move that looks exactly like a move created by two different people. The combination creates a new kind of redundancy that didn't exist before.

6. Training and "Conserved Quantities"

Finally, the paper touches on how these networks learn (using gradient flow).

In some cases, these symmetries act like conservation laws in physics (like energy or momentum). As the network trains, certain mathematical quantities stay exactly the same, no matter how the numbers change.
However, the authors found that the new "hiding" symmetries they discovered do not create these conservation laws. This means the network's training path behaves differently depending on which type of symmetry is active.

Summary

The paper provides a complete "user manual" for the hidden symmetries in three-layer ReLU networks. It reveals that:

Layers can hide each other: One layer can make the geometric features of a previous layer invisible to the rest of the network.
This creates infinite variations: You can shift or flip these "hidden" features without changing the network's output.
We can detect it: There is a fast way to tell if two networks are functionally identical.
It's a global property: You can't just check layer by layer; you have to look at how the whole machine fits together to see these hidden redundancies.

The authors conclude that while they have solved this for "bottleneck" networks, making the network wider or deeper makes the problem exponentially harder, likely hitting a wall of computational complexity.

Technical Summary: The Symmetries of Three-Layer ReLU Networks

Problem Statement
A fundamental property of neural network parameterizations is redundancy: distinct parameter configurations can realize the exact same function. These redundancies are formalized as fibers, defined as the set of all parameters realizing a given function. While parameter symmetries (transformations leaving the function unchanged) are well-understood for single-hidden-layer ReLU networks—where generic fibers are trivial up to permutation and scaling—the structure of fibers in deep architectures remains largely uncharacterized. In deep networks, additional local symmetries can arise from specific non-degenerate configurations, and identifiability is often non-uniform across the parameter space. The central problem addressed is the systematic description of all parameters realizing a given function in deep ReLU networks, specifically investigating how layer composition introduces redundancies that cannot be reduced to symmetries acting within individual layers.

Methodology
The authors develop a framework based on polyhedral geometry and tropical geometry to analyze parameter fibers. Key methodological components include:

Hyperplane Representation: For one-hidden-layer networks on the nonnegative orthant, the realized function is characterized by a weighted hyperplane arrangement. The function is determined up to a global affine transformation by the arrangement of "bent hyperplanes" (where neurons switch from inactive to active) and their associated weight vectors (gradient jumps).
Genericity Assumptions: The analysis focuses on generic parameters, defined by two conditions: (1) Supertransversality, ensuring bent hyperplanes intersect in expected codimensions, and (2) Maximal rank of masked products, preventing algebraic degeneracies in weight matrix compositions. This ensures the combinatorial structure of the canonical polyhedral complex is stable.
Layerwise vs. Compositional Analysis: The paper first characterizes fibers for single layers (layerwise symmetries) and then analyzes how these interact when layers are composed. A critical mechanism identified is the "hiding" of geometric features: a hyperplane from an earlier layer may be functionally invisible in the final output if it is "anchored" or covered by the active regions of subsequent layers.
Semi-Algebraic Descriptions: The fibers are described explicitly as unions of semi-algebraic sets defined by polynomial equations and inequalities, allowing for dimension calculations and algorithmic decision procedures.

Key Contributions and Results

Layerwise Fibers (One Hidden Layer):
The paper provides a complete description of fibers for one-hidden-layer networks on the nonnegative orthant. The fiber is shown to be a disjoint union of semi-algebraic sets corresponding to different orientation choices of the nonlinear neurons. Once orientations are fixed, the remaining degrees of freedom correspond to the factorization of an affine map by linear neurons. The dimension of these components is computed based on the nonnegative column rank of the required linear part.
Symmetries from Layer Composition (Three-Layer Networks):
The authors identify new symmetries arising specifically from the composition of ReLU layers, which are not present in single-layer analysis.
- Continuous Symmetry: If a second-layer neuron "hides" a first-layer hyperplane (by being active only when the first-layer neuron is linear and inactive elsewhere), the first-layer bias can be translated continuously. This translation is exactly compensated by a corresponding shift in the second-layer biases, leaving the realized function invariant.
- Discrete Symmetry: In the specific case where the second hidden layer consists of a single neuron ( $n_2=1$ ), a discrete "sign-flipping" symmetry emerges. This allows for a finite group of transformations on the first-layer weights that preserve the function, distinct from standard permutation and scaling.
Complete Characterization for Three-Layer Bottlenecks:
For three-layer bottleneck architectures ( $n_1 \leq d$ ), the paper proves that the symmetries described above (permutation, scaling, layerwise factorization, hiding-induced translation, and discrete sign-flipping) are complete for generic parameters.
- Identifiability Conditions: A generic parameter is identifiable if and only if no first-layer hyperplane is hidden. This condition is determined entirely by the sign patterns of the second-layer weights and biases. Specifically, non-identifiability occurs if a row of the second-layer weight-bias matrix is all negative (dead neuron), all positive (linear neuron), or has a specific "hiding" pattern (positive in one column, negative elsewhere).
- Polynomial-Time Algorithm: Based on this characterization, the authors provide a polynomial-time algorithm to decide whether two generic parameters in a bottleneck architecture realize the same function.
Non-Localizability of Fibers:
The paper demonstrates that identifiability in deep ReLU networks is not localizable. It constructs a network where pairwise compositions of layers ( $f_2 \circ f_1$ and $f_3 \circ f_2$ ) are identifiable, yet the full composition ( $f_3 \circ f_2 \circ f_1$ ) is not. This shows that parameter redundancy can arise as a global property of network composition, dependent on the interaction between the range of earlier layers and the non-linearities of later ones.
Conservation Laws in Gradient Flow:
The authors distinguish between symmetries that induce conserved quantities along gradient flow and those that do not.
- Symmetries arising from linear neurons (layerwise scaling) induce locally conserved quantities (related to matrix product symmetries).
- Conversely, the continuous "hiding" symmetries identified in Section 5 do not induce conserved quantities, as the symmetry curve is not orthogonal to the gradient flow in a way that generates a potential function.

Significance and Claims
The paper claims to provide the first systematic and complete description of parameter fibers for deep ReLU networks beyond the single-hidden-layer case, specifically for the bottleneck regime. By establishing that fiber structures depend on whether deeper layers hide or preserve geometric structure from preceding layers, the work clarifies the nature of parameter redundancy in deep learning.

The authors position their results near a tractability boundary. They argue that the explicit, efficiently checkable descriptions of fibers are possible specifically because of the genericity and bottleneck assumptions. They note that removing these assumptions (e.g., in wider or deeper networks) leads to combinatorial blow-ups and is consistent with known complexity-theoretic barriers (coNP-hardness) for deciding functional equivalence in general ReLU networks. Thus, the paper isolates a regime where genuinely deep compositional symmetries arise but remain analytically tractable.