Scaling Laws and Pathologies of Single-Layer PINNs: Network Width and PDE Nonlinearity

This paper establishes that Single-Layer Physics-Informed Neural Networks suffer from dual optimization pathologies—baseline and compounding failures driven by spectral bias and PDE nonlinearity—that prevent error reduction with increased width, revealing that optimization, rather than approximation capacity, is the primary bottleneck governed by complex, non-separable scaling laws.

The Gist

The Big Picture: The "Wider is Better" Myth

In the world of Artificial Intelligence, there is a popular rule of thumb: "If you make the neural network wider (add more neurons), it gets smarter and solves problems better." This works great for things like recognizing cats in photos or translating languages.

However, this paper investigates a specific type of AI called a PINN (Physics-Informed Neural Network). These are AIs trained to solve complex math equations that describe how the physical world works (like how water flows or how heat spreads).

The author, Faris Chaudhry, discovered a shocking truth: For these physics problems, making the network wider often makes it worse, or at best, does nothing.

The Core Problem: The "Spectral Bias" Traffic Jam

Imagine you are trying to paint a picture of a landscape.

• The Easy Part: Painting the big blue sky and the green hills (low-frequency, smooth shapes).
• The Hard Part: Painting the tiny, jagged details of a rocky cliff or the ripples on a stream (high-frequency, sharp details).

Neural networks have a natural quirk called Spectral Bias. It's like an artist who is great at painting big, smooth blobs but terrible at painting tiny, sharp details. They naturally ignore the "noise" and focus on the "smoothness."

When the physics problem is simple (like a smooth hill), the AI does fine. But when the problem gets nonlinear (meaning the physics gets chaotic, like a stormy ocean or a shockwave), the solution requires those tiny, sharp details. The AI's "Spectral Bias" causes it to get stuck. It keeps painting the smooth sky and completely misses the jagged rocks.

The Two "Pathologies" (The Double Trouble)

The paper identifies two specific ways this AI fails:

1. The Baseline Pathology: "More Brains, Same Confusion"

You might think, "If the AI is confused, let's just give it more neurons (make the network wider) so it has more brainpower to figure it out."

• The Reality: The author found that for these physics problems, adding more neurons is like giving a confused person a bigger library. They still don't know how to read the right book.
• The Result: Even with a massive network, the error doesn't go down. In fact, sometimes it goes up. The AI isn't failing because it lacks "capacity" (brainpower); it's failing because the training process (how it learns) is broken. It's stuck in a local trap and can't find the right path, no matter how wide the net is.

2. The Compounding Pathology: "The Chaos Multiplier"

This is where it gets worse. The author tested problems with different levels of "hardness" (nonlinearity).

• The Analogy: Imagine trying to walk across a room.
  • Low Hardness: The floor is flat. You can walk easily.
  • High Hardness: The floor is covered in slippery ice and moving obstacles.
• The Finding: As the problem gets "slipperier" (more nonlinear), the AI's ability to learn collapses. The relationship between "Network Width" and "Success" breaks down completely. You can't just use a simple formula (like Success = Width × Difficulty) to predict the outcome. The difficulty changes the rules of the game entirely.

The Experiments: Testing the Limits

The author tested this on three famous physics equations:

1. KdV (Water Waves): Testing how big a wave (soliton) is.
2. Sine-Gordon (Waves in a String): Testing how strong the non-linear pull is.
3. Allen-Cahn (Phase Changes): Testing how sharp the boundary is between two states (like ice and water).

The Results:

• Linear Problems (Smooth): The AI struggled a bit, but it could learn.
• Nonlinear Problems (Chaotic): The AI failed spectacularly.
  • Using ReLU (a common activation function) was like trying to draw a smooth curve with a jagged ruler; it failed completely.
  • Using Tanh (a smoother function) was slightly better but still couldn't overcome the "traffic jam" of learning.
  • Crucially: Making the network wider (from 16 neurons to 1024 neurons) did not fix the problem. The error stayed high or got worse.

The Takeaway: Stop "Brute-Forcing" It

The main conclusion is a wake-up call for researchers.

• Don't just build bigger, wider networks. For physics problems, throwing more computing power at a simple, wide, single-layer network is a waste of time.
• The bottleneck isn't the size of the brain; it's the method of learning. The AI is trying to learn the "high-frequency" details (the jagged rocks) but its learning algorithm is biased toward the "low-frequency" details (the smooth sky).
• The Future: We need new ways to train these networks (better optimizers, different architectures, or adding "Fourier features" to help them see the high-frequency details) rather than just making them wider.

In short: If you are trying to teach an AI to solve complex physics equations, simply giving it a bigger brain won't help if it's using the wrong learning strategy. It's not about how big the network is; it's about how well it can see the sharp, chaotic details of the physical world.

Technical Summary

1. Problem Statement

The paper addresses a critical gap between the theoretical expressivity of Physics-Informed Neural Networks (PINNs) and their practical optimization performance.

• Theoretical Expectation: The Universal Approximation Theorem (UAT) and approximation theory (e.g., Barron spaces) suggest that even Single-Layer Networks (SLNs) should approximate continuous functions with an error scaling of $O(N^{-1/2})$ as network width ($N$) increases.
• Practical Reality: PINNs often fail to solve complex Partial Differential Equations (PDEs), particularly those with high nonlinearity.
• Core Question: Does increasing network width improve solution accuracy in PINNs as theory predicts, or do optimization challenges (specifically spectral bias and non-convex loss landscapes) prevent this? The authors investigate whether the scaling relationship is separable (independent effects of width and nonlinearity) or non-separable (coupled effects).

2. Methodology

The authors propose an empirical framework to derive scaling laws for Single-Layer PINNs across a suite of canonical nonlinear PDEs.

• Architecture: Single-Layer Neural Networks (SLNs) with varying widths ($N \in \{16, \dots, 1024\}$).
• Activation Functions: Both Tanh (smooth, $C^\infty$) and ReLU (non-smooth) are tested to isolate the impact of spectral bias and derivative representation.
• PDE Suite & Hardness Parameter ($\kappa$):
  • Poisson Equation (Linear): Used as a baseline to validate theoretical scaling rates.
  • Korteweg-de Vries (KdV): Dispersive; $\kappa$ controls soliton amplitude/sharpness.
  • Sine-Gordon: Hyperbolic/Transcendental; $\kappa$ scales the nonlinear potential.
  • Allen-Cahn: Reactive/Parabolic; $\kappa = 1/D$ (inverse diffusion), controlling interface sharpness.
  • Note: In all nonlinear cases, increasing $\kappa$ increases the high-frequency components of the ground truth solution, challenging the network's spectral bias.
• Training Protocol:
  • Loss: Weighted sum of PDE, Boundary Condition (BC), and Initial Condition (IC) residuals (equal weights).
  • Optimizer: Adam ($lr=10^{-3}$) for 25,000 epochs.
  • Evaluation: Mean relative $L_2$ error against high-fidelity analytical/numerical solutions.
• Scaling Analysis:
  1. Univariate: Fit error $\approx A \cdot N^{-\alpha}$ at fixed $\kappa$ to determine the width exponent $\alpha(\kappa)$.
  2. Multivariate: Fit separable power law $Error \approx A \cdot N^{-\alpha} \cdot \kappa^{\gamma}$ and compare against non-separable interaction models.

3. Key Contributions

1. Identification of Dual Optimization Failure: The paper establishes that PINN failure is not due to a lack of approximation capacity but a failure of optimization, characterized by two pathologies:
   • Baseline Pathology: Error fails to decrease with width even at fixed nonlinearity ($\alpha \approx 0$ or negative), falling short of the theoretical $\alpha = 0.5$.
   • Compounding Pathology: Nonlinearity exacerbates this failure, causing the simple separable scaling law to break down.
2. Quantification of Non-Separable Scaling: The authors provide evidence that the scaling exponent $\alpha$ is a function of the nonlinearity $\kappa$. The relationship is non-separable, meaning the effect of width depends heavily on the problem's hardness.
3. Activation Function Divergence: A distinct mechanistic difference is found between ReLU and Tanh networks regarding how they interact with nonlinearity (specifically, ReLU shows statistically significant non-separable interactions, while Tanh does not).

4. Key Results

• Linear Benchmark (Poisson):
  • Tanh: Converges to low error ($\approx 10^{-3}$) but exhibits high variance and no consistent scaling trend ($\alpha \approx 0.06$).
  • ReLU: Catastrophic failure. Error remains high ($\approx 1.0$) regardless of width ($\alpha \approx 0.01$). This is attributed to spectral bias; ReLU's second derivative is a sparse set of Dirac deltas, making it ill-suited for the smooth derivatives required by the PDE loss.
• Nonlinear PDEs (KdV, Sine-Gordon, Allen-Cahn):
  • Width Scaling ($\alpha$): Consistently near-zero or negative across all nonlinear PDEs. Increasing width often increases the error, contradicting the "wider is better" heuristic of deep learning.
  • Hardness Scaling ($\gamma$): Generally positive (error increases with $\kappa$), except for Allen-Cahn with ReLU (negative $\gamma$), suggesting a qualitatively different failure mechanism.
  • Dominance of Nonlinearity: Changes in hardness $\kappa$ alter error by several orders of magnitude, whereas changes in width $N$ typically result in less than an order of magnitude difference.
• Separability Test:
  • The simple separable model ($N^{-\alpha}\kappa^{\gamma}$) is insufficient.
  • Interaction: For ReLU, the interaction term between width and nonlinearity is statistically significant, confirming a non-separable relationship. For Tanh, width ceases to be a statistically significant factor entirely at higher nonlinearities.

5. Significance and Implications

• Optimization vs. Capacity: The primary bottleneck for PINNs is optimization, not approximation capacity. The complex, non-convex loss landscapes induced by nonlinear PDEs prevent gradient-based optimizers from finding the theoretical minima, regardless of network size.
• Spectral Bias: The results reinforce that spectral bias (the tendency to learn low frequencies first) is a fundamental limitation. As nonlinearity increases, the solution requires high-frequency components that standard PINNs struggle to learn, leading to "pathological scaling" where wider networks perform worse.
• Inefficiency of "Brute Force": Simply increasing the width of shallow PINNs is an inefficient strategy for solving complex nonlinear PDEs.
• Future Directions: The paper calls for scaling studies on more advanced architectures (multi-layer, Fourier features, attention mechanisms) and optimizers (adaptive weighting, second-order methods) that can mitigate spectral bias and close the gap between theoretical and empirical scaling.

In summary, this work provides a rigorous quantitative framework demonstrating that standard Single-Layer PINNs suffer from fundamental optimization pathologies that render simple width-scaling ineffective for nonlinear problems, necessitating a shift in architectural and training strategies.