📊 statistics

Ghosts in Neural Networks: Existence, Structure and Role of Infinite-Dimensional Null Space

This paper establishes a direct method for solving the neural network synthesis equation in continuous-width depth-two networks by leveraging Fourier analysis and ridgelet transforms to characterize the infinite-dimensional null space, identify unique minimum-norm solutions, and demonstrate how parameter nonuniqueness can be discretized and exploited to reveal encoded information.

Original authors: Sho Sonoda, Isao Ishikawa, Masahiro Ikeda

Published 2026-07-21

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Sho Sonoda, Isao Ishikawa, Masahiro Ikeda

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a machine that can predict the future, or perhaps just recognize a cat in a photo. You build a "neural network," which is basically a giant web of tiny mathematical switches called neurons. Each neuron takes some input, does a little calculation, and passes the result along. The magic happens when you adjust the "knobs" (called parameters) on these neurons to make the machine output the right answer.

But here's a weird thing about these machines: sometimes, you can twist the knobs in a completely different way, and the machine still gives you the exact same answer. It's like if you were baking a cake and found that adding a cup of sugar or a cup of salt (in a very specific, weird recipe) made the cake taste exactly the same. In math, we call this "non-uniqueness." It means there isn't just one perfect set of knobs to get a result; there are infinitely many.

This paper dives deep into that mystery. It asks: If there are infinite ways to set the knobs to get the same result, what do those "wrong" settings actually look like? Do they just vanish into nothingness, or are they hiding something? The authors treat the neural network like a giant, continuous wave rather than a bunch of separate switches. By using a mathematical tool called the "Fourier transform" (think of it as a way to break a complex sound into its individual musical notes), they figure out exactly how these extra, hidden settings work. They discover that these hidden settings form a giant, invisible "ghost" space. You can add these ghosts to your machine, and it won't change the output at all, but it might change how the machine behaves if you poke it or look at it differently.

The Ghosts in the Machine

The paper, titled "Ghosts in Neural Networks," is a deep dive into the hidden secrets of two-layer neural networks. The authors, Sho Sonoda, Isao Ishikawa, and Masahiro Ikeda, are essentially solving a giant puzzle: If a neural network can produce a specific output in many different ways, what are all those different ways?

They found that the answer isn't just "random noise." Instead, there is a structured, mathematical "ghost" space (called the null space) where you can hide infinite amounts of information without changing the final result.

The "Ghost" Analogy

Imagine you have a magical painting machine. You give it a command: "Paint a red circle."

The Standard Way: You tell the machine to mix red paint and draw a circle.
The Ghost Way: You could tell the machine to mix red paint, draw a circle, and then add a secret, invisible layer of "anti-red" paint that perfectly cancels itself out, or add a secret swirl of invisible ink that does absolutely nothing to the final picture.

In the world of neural networks, these "invisible layers" are the ghosts. The paper proves that these ghosts aren't just random mistakes; they have a very specific shape and structure. The authors call this the infinite-dimensional null space. It's a giant, invisible room where you can store infinite variations of the machine's settings, and as long as you stay in that room, the machine's output (the red circle) never changes.

How They Found the Ghosts

The authors didn't just guess; they built a direct mathematical map to find these ghosts.

Breaking it Down: They used a technique called "separation of variables." Imagine you have a complex recipe. Instead of trying to taste the whole dish at once, you separate the ingredients: "This part is the flour, this part is the sugar." They separated the neural network's math into two parts: the part that actually creates the image (the target) and the part that does nothing (the ghost).
The Ridgelet Solution: They found a specific mathematical formula, called a ridgelet, that acts like a key to unlock these ghosts. It's like having a master key that can open any door in the "ghost room."
The Minimum-Norm Solution: Among all the infinite ways to set the knobs, there is one "perfect" way that uses the least amount of energy (mathematically, the "minimum norm"). The authors showed how to find this specific, most efficient setting. Everything else is just a "ghost" added on top.

What This Means for Real Networks

The paper doesn't just stop at theory; it shows how these ghosts behave in the real world.

Finite Width: Real neural networks aren't infinite; they have a limited number of neurons (a "finite width"). The authors showed that even with a limited number of neurons, you can still approximate these ghosts. If you take a "ghost" setting and break it down into a finite number of neurons, the machine's output will be almost zero (very close to the ghost), with an error that shrinks as you add more neurons. It's like trying to draw a perfect circle with a limited number of dots; the more dots you use, the closer you get to a perfect circle.
Numerical Proof: They actually ran computer simulations to prove this. They created a "ghost" setting and fed it into a finite network. The result? The network output was incredibly close to zero, confirming that the ghosts exist even in small, real-world networks.
Reading the Ghosts: Here is the coolest part. The authors showed that even though these ghosts don't change the output under normal conditions, you can "read" them if you change the rules slightly. If you tweak the machine's activation function (the rule the neurons follow) just a tiny bit, you can make the hidden ghost information pop out and become visible. It's like having a secret message written in invisible ink that only appears when you hold the paper up to a specific kind of light.

What They Don't Claim

It's important to know what this paper doesn't say.

It doesn't say that every single finite network has a giant empty space of ghosts. Sometimes, if the neurons are arranged in a specific, rigid way, the ghosts might disappear entirely. The paper clarifies that the "ghost room" is a property of the continuous (infinite) version of the network, and finite networks only approximate these ghosts.
It doesn't say that learning algorithms (like the ones that train AI) automatically find these ghosts. The paper proves the ghosts exist and shows how to find them mathematically, but it doesn't claim that a computer learning to play chess will naturally stumble upon them. That's a separate question for future research.

The Big Takeaway

This paper is a mathematical tour de force that turns a confusing problem ("Why are there so many ways to set the knobs?") into a clear, structured map. It proves that the "extra" settings in a neural network aren't just random noise; they are a structured, infinite library of hidden possibilities.

By finding the "Ridgelet" key, the authors showed us how to:

Find the most efficient setting (the minimum-norm solution).
Understand the infinite "ghost" settings that don't change the output.
Approximate these ghosts in real, finite networks.
Potentially read the hidden information if we know how to poke the system just right.

In short, the paper reveals that neural networks are far more flexible and mysterious than we thought. They aren't just simple calculators; they are vast, multi-dimensional spaces where you can hide infinite secrets, and with the right mathematical tools, you can find them all.

Technical Summary: Ghosts in Neural Networks

Problem Statement

The paper addresses the problem of parameter nonuniqueness in continuous-width, depth-two fully connected neural networks. Specifically, it seeks to solve the "neural-network equation" $S[\gamma] = f$ , where $S$ is the synthesis operator mapping a parameter distribution $\gamma$ (over weights $a$ and biases $b$ ) to a network function $f$ .

While the linearity of the map $\gamma \mapsto S[\gamma]$ allows for the study of neural parameters via operator theory, it also implies redundancy: distinct parameter distributions can represent the exact same function. The central challenge is to characterize the complete solution set, including the null space (or "ghosts") of the operator $S$ , and to identify a canonical, minimum-norm solution. Previous work had identified that ridgelet functions orthogonal to the activation generate null elements, but a rigorous, exhaustive characterization of the entire null space and a direct method for solving the equation without postulating a transform were lacking.

Methodology

The authors employ a direct Fourier expression method combined with functional analysis in Hilbert spaces. The approach proceeds in three logical stages:

Fourier Separation of Variables:
Instead of assuming a ridgelet transform, the authors begin with the Fourier expression of the network integral. By performing Fourier inversion on the preactivation variable and separating the target frequency $\xi$ from the one-dimensional frequency $\omega$ , they derive a parameter formula. This separation suggests a specific structure for the parameter distribution $\gamma$ that inverts the synthesis operator, naturally leading to a ridgelet solution.
Abstract Reconstruction and Geometry:
To rigorize the formal Fourier calculation, the paper isolates the underlying algebraic mechanism: a unitary factorization of the synthesis operator. The authors prove an abstract reconstruction theorem for unitary factorizations, establishing the existence of an adjoint, a normalized right inverse, and an orthogonal decomposition of the solution space. This framework defines the geometry of all solutions as a direct sum of a particular solution and the null space.
Hilbert Space Specialization:
The abstract theory is specialized to neural networks by constructing compatible Hilbert spaces for the activation, coefficients, and parameters.
- Activation Space ( $A_{s,t}$ ): A weighted Sobolev space accommodating tempered distributions (e.g., ReLU, sigmoid) with polynomial growth.
- Coefficient Space ( $H_{s,t}$ ): A space ensuring the activation functional is bounded and continuous.
- Parameter Space ( $G_{s,t}$ ): A graph space defined by the unitary coordinate transform, ensuring the synthesis operator $S: G_{s,t} \to L^2(\mathbb{R}^m)$ is bounded.
  This construction allows the formal Fourier steps to be justified analytically, proving that the synthesis operator is bounded and that limits can be passed through the integral.

Key Contributions and Results

1. Direct Solution and Null Space Characterization

The primary contribution is a direct derivation of the ridgelet solution from the Fourier expression of the neural-network equation.

Particular Solution: The method yields a specific ridgelet transform $R[f; \rho_0]$ that solves $S[\gamma] = f$ .
Complete Null Space: The paper proves that the null space is exhausted by a Hilbert-basis expansion of ridgelet functions $R[e_i; \rho_j]$ where the pairing $\langle \sigma, \rho_j \rangle = 0$ .
General Solution: Every solution $\gamma$ is uniquely represented as the sum of the minimum-norm particular solution and an arbitrary element from the null space.

2. Canonical Minimum-Norm Solution

By constructing the Hilbert adjoint $S^*$ and the Moore-Penrose inverse $S^\dagger$ , the authors identify a unique minimum-norm parameter distribution for any target function $f$ . This solution is given by $S^\dagger f = c_\sigma^{-1} R[f; \sigma^*]$ , where $\sigma^*$ is the adjoint ridgelet function (the Riesz representer of the activation functional).

3. Explicit Adjoint Ridgelet Functions

The paper provides concrete formulas for the adjoint ridgelet function $\sigma^*$ for standard activations (ReLU, tanh, Gaussian CDF, and Gaussian density) within specific Hilbert geometries. These are derived as solutions to one-dimensional resolvent problems in the Fourier domain.

4. Finite-Width Approximation and Discretization

The authors investigate how continuous null elements relate to finite-width networks:

Approximate Nulls: They prove that any nonzero null measure can be discretized into a finite-width network of width $N$ with an output error of $O(N^{-1/2})$ . This establishes that parameter redundancy is not merely an infinite-width pathology but leaves a trace in finite networks.
Exact Nulls: Exact finite null relations are shown to arise only from specific algebraic symmetries of the activation (e.g., parity for odd/even functions or affine cancellation for ReLU), rather than from the general continuous null space.

5. Perturbative Readout

The paper demonstrates that information encoded in the null space (which is invisible to the standard activation $\sigma$ ) can be made visible by applying a specific additive parameter perturbation. This perturbation, which is the unique minimum-norm correction, allows a "decoder" activation to read out functions stored in the null component without altering the null component itself.

Significance and Scope

The paper claims to provide the first rigorous, exhaustive solution to the neural-network equation for continuous-width depth-two networks using a direct Fourier method. Its significance lies in:

Rigorous Foundation: It replaces heuristic ridgelet constructions with a bounded operator theory framework, proving that the Fourier separation-of-variables method is mathematically sound for tempered-distribution activations.
Structural Clarity: It explicitly separates the "visible" target function from the "ghost" null components, providing a complete geometric description of the parameter space.
Canonical Selection: It offers a principled way to select a unique minimum-norm parameter distribution, resolving the ambiguity inherent in over-parameterized networks.

The authors explicitly state that the work focuses on the functional analysis of the synthesis operator and the complete solution of the equation. While the results have potential implications for learning theory (e.g., generalization bounds, neural tangent kernels), the paper treats these as consequences to be investigated separately rather than claims of the current structure theorem. The results are formalized in Lean 4 to ensure correctness.