🤖 machine learning

Can an MLP Absorb Its Own Skip Connection?

The paper demonstrates that for most common activation functions (such as ReGLU, SwiGLU, and GELU) and generic weight distributions, a skip connection cannot be absorbed into a residual-free MLP of the same width, meaning skip-connected and residual-free architectures represent fundamentally different function classes.

Original authors: Antonij Mijoski, Marko Karbevski

Published 2026-04-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Antonij Mijoski, Marko Karbevski

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Question: Can a "Shortcut" be Hidden Inside the Machine?

Imagine you are building a complex LEGO castle. Most of the time, you follow a strict set of instructions: Step 1: Build a wall. Step 2: Build a tower. Step 3: Build a gate. This is how a standard neural network (an MLP) works—it processes information in a straight line, layer by layer.

However, modern AI (like the tech behind ChatGPT) uses a "cheat code" called a Skip Connection. Instead of just going from Step 1 to Step 2, the instructions say: "Build a wall, but also keep a copy of the original baseplate and just slide it forward to the next step." This "shortcut" helps information flow more easily and prevents the "instructions" from getting lost or corrupted as the castle gets taller.

The researchers asked a fascinating question: If we have this "shortcut" (the skip connection), is it actually doing something special? Or could we just rewrite the instructions so that the shortcut is "absorbed" into the building process itself? In other words: Could we build the exact same castle without the shortcut, just by changing how we build the walls and towers?

The Verdict: The Shortcut is Magic

The researchers looked at different types of "building materials" (called activations) that AI uses, and they found that, in almost every case, the shortcut is indispensable. You cannot hide it.

Here is how they broke it down using three different scenarios:

1. The "Math-Heavy" Materials (Gated Activations)

The Analogy: The Shape-Shifting Clay.
Some modern AI models use very complex, "gated" materials (like SwiGLU or GeGLU). Think of these like a special kind of clay that changes its shape and density depending on how much pressure you apply.

The researchers proved that because this clay is so "reactive," it is mathematically impossible to mimic the shortcut. If you try to build a castle without the shortcut using this clay, the math simply won't balance. The shortcut isn't just a convenience; it’s a fundamental part of the architecture that the clay cannot replicate.

2. The "Simple" Materials (ReLU and GELU)

The Analogy: The Rigid Bricks.
Other models use simpler materials, like ReLU (which acts like a light switch: it's either ON or OFF). For these, the researchers found a tiny, tiny loophole.

It is technically possible to absorb the shortcut, but only if you arrange your bricks in a very specific, almost impossible way (a "measure-zero" event). It’s like saying, "You can build this castle without a shortcut, but only if every single brick is placed at a perfect 45.0000000001-degree angle." In the real world, where weights are chosen somewhat randomly, this will never happen. For all practical purposes, the shortcut is a unique superpower.

3. The "Deep" Problem (Deep Networks)

The Analogy: The Infinite Skyscraper.
Finally, they looked at what happens when you stack these layers hundreds of times to make a skyscraper. They proved that even if you could somehow "cheat" and hide a shortcut in one single floor, that trick won't work as you go higher. The "shortcut" effect compounds. By the time you reach the 100th floor, the difference between a building with shortcuts and a building without them is massive.

Why Does This Matter?

If the shortcut could be absorbed, it would mean that skip connections are just a "math trick" to make training easier, but they don't actually make the AI smarter.

But because the researchers proved they cannot be absorbed, it tells us something profound: Skip connections actually expand the "brain power" of the AI. They allow the network to represent types of logic and patterns that a standard, straight-line network simply cannot grasp.

The takeaway: The "shortcuts" in modern AI aren't just helping the math flow more smoothly; they are actually giving the AI a bigger, more complex world to think in.

Technical Summary: "Can an MLP Absorb Its Own Skip Connection?"

Authors: Antonij Mijoski and Marko Karbevski
Core Research Question: Does the addition of a skip connection (residual connection) to a Multi-Layer Perceptron (MLP) expand its representational capacity, or is it merely a reparameterization that could be "absorbed" into a standard, residual-free MLP of the same width?

1. The Problem

In modern deep learning (Transformers, LLaMA, etc.), almost every MLP layer is wrapped in a residual connection: $f(x) = x + \text{MLP}(x)$ . While it is well-known that skip connections improve optimization (gradient flow and loss landscape smoothing), the authors investigate the algebraic redundancy of this structure.

Specifically, they ask: Given a skip-connected MLP $\text{MLP}_{\text{skip}}(x) = x + W_{\text{down}}\sigma(W_{\text{up}}x)$ , does there exist a residual-free MLP $\text{MLP}_{\text{ff}}(x) = V_{\text{down}}\sigma(V_{\text{up}}x)$ of the same width such that $\text{MLP}_{\text{skip}}(x) = \text{MLP}_{\text{ff}}(x)$ for all $x$ ?

2. Methodology

The authors employ a rigorous mathematical approach combining functional analysis, parity decomposition, and measure theory. They categorize activation functions into three distinct groups to test the limits of absorption:

Homogeneous activations: Functions where $\sigma(\lambda z) = \lambda^k \sigma(z)$ (e.g., $\text{ReLU}^2$ , $\text{ReGLU}$ ).
Gated activations: Modern architectures where the activation is a product of a gate and a value (e.g., $\text{SwiGLU}$ , $\text{GeGLU}$ ).
Ungated activations: Standard activations like $\text{ReLU}$ and $\text{GELU}$ .

They use parity decomposition (splitting a function into its even and odd components) to analyze how the linear term $x$ (the skip connection) interacts with the non-linear components of the MLP.

3. Key Contributions and Results

A. Impossibility for Modern State-of-the-Art (SOTA) Activations
The paper proves that for the activations used in most current LLMs, absorption is unconditionally impossible.

Homogeneous Activations (Prop 5): For activations of degree $k \neq 1$ (like $\text{ReLU}^2$ ), a degree argument shows that a linear map and a degree- $k$ MLP live in incompatible homogeneity classes.
Gated Activations (Prop 6 & Cor 7): For gated activations like $\text{SwiGLU}$ and $\text{GeGLU}$ , a linearization argument shows that the gated MLP output is $O(\|x\|^2)$ near the origin. Since the skip connection is $O(\|x\|)$ , they can never be equal. This result extends to arbitrary depth, meaning deep residual networks using these activations cannot be replicated by deep residual-free networks of the same width.

B. The "Special Case" of Ungated ReLU and GELU
For standard $\text{ReLU}$ and $\text{GELU}$ , absorption is mathematically possible but non-generic.

The Submatrix Condition (Theo 8 & 10): Absorption occurs if and only if there exists a subset of neurons $S$ (where $|S| \geq d$ ) such that the product of the down-projection and up-projection weights for those neurons satisfies: $W_{\text{down}}[:, S] W_{\text{up}}[S, :] = -\text{Id}$ .
Generic Disjointness (Cor 11): Because this condition requires a very specific algebraic alignment, it occurs with Lebesgue measure zero. In other words, for almost any randomly initialized weight matrix, a skip-connected MLP and a residual-free MLP represent entirely different function classes.

C. Generalization to Learnable Skips
The authors prove (Lemma 4) that if the skip connection is a learnable invertible linear map (as in "Hyper-Connections"), the problem reduces to the identity skip case, meaning their impossibility results still hold.

4. Significance

This paper provides a formal theoretical foundation for why skip connections are not just an optimization trick, but a structural necessity for representational power.

Representational Expansion: It proves that skip connections grant access to a region of the function space that a standard MLP of the same width cannot reach.
Architectural Design: The results justify the ubiquity of residual connections in Transformers. If we were to remove them, we could not simply "re-weight" the existing MLP layers to compensate; we would fundamentally be changing the model's capacity.
Theoretical Boundary: The paper identifies a sharp divide: while $\text{ReLU}$ and $\text{GELU}$ allow for rare, highly specific instances of absorption, the gated activations that power the current AI revolution ( $\text{SwiGLU}$ , etc.) make skip connections algebraically indispensable.