A new initialisation to Control Gradients in Sinusoidal Neural network

Technical Summary: A New Initialisation to Control Gradients in Sinusoidal Neural Networks

Problem Statement
Implicit Neural Representations (INRs), particularly those utilizing sinusoidal activation functions (SIREN), are effective for approximating high-frequency signals in applications such as image reconstruction, signal encoding, and solving partial differential equations (PDEs). However, standard SIREN architectures face a critical trade-off: while they mitigate the spectral bias of standard Multi-Layer Perceptrons (MLPs) by learning high frequencies, increasing network depth often leads to the emergence of spurious high-frequency artifacts. This phenomenon degrades generalization and causes gradient explosion. Conversely, standard initialization schemes (e.g., Xavier, Kaiming) or the original SIREN initialization fail to provide theoretical guarantees for gradient stability in deep periodic networks, often resulting in vanishing or exploding gradients that hinder training dynamics.

Methodology
The authors propose a new, analytically derived initialization scheme for SIREN networks designed to operate at the "edge of chaos" (EOC). This regime ensures that pre-activation variances remain stable and gradients neither vanish nor explode as network depth increases.

The methodology relies on a theoretical analysis in the limit of infinite width ($N \to \infty$) and infinite depth ($L \to \infty$):

1. Pre-activation Distribution: The authors derive a closed-form expression for the fixed point of the pre-activation variance ($\sigma_a^2$) using the Lambert $W$ function. They show that the variance converges to a fixed point determined by the weight scaling parameter $c_w$ and bias variance $c_b$.
2. Gradient Stability: By analyzing the Jacobian of the layer-to-layer mapping, the authors derive the condition for gradient stability. They define a rescaled gradient variance $\sigma_g$ and enforce $\sigma_g = 1$ to prevent gradient explosion or vanishing. This constraint establishes a specific relationship between $c_w$ and $c_b$.
3. Spectral Control: The authors identify two specific solutions along the stability curve:
   • $\sigma_a = 1$: Corresponds to the original SIREN intuition (pre-activations follow $N(0,1)$), which the authors show leads to spectral broadening and aliasing in deep networks.
   • $\sigma_a = 0$ (Proposed): Sets the pre-activation variance to zero. In this regime, deep layers behave almost linearly ($\sin(z) \approx z$), effectively suppressing the broadening of the Fourier spectrum and preventing the emergence of frequencies beyond the Nyquist limit.
4. Neural Tangent Kernel (NTK) Analysis: The paper investigates the initialization's impact on the NTK. They demonstrate that the proposed initialization ($\sigma_a=0$) results in NTK eigenvalues that scale linearly with depth, while input gradients remain normalized. This contrasts with the original SIREN initialization, where NTK eigenvalues and gradients explode exponentially with depth.

Key Contributions

• Closed-Form Initialization: The derivation of an explicit initialization scheme for SIREN that guarantees invariant gradient distributions across layers and a depth-independent Fourier spectrum.
• Theoretical Justification for $\sigma_a = 0$: The identification that setting the pre-activation variance to zero is optimal for controlling the frequency spectrum, preventing aliasing, and ensuring stable training dynamics in deep architectures.
• NTK Dynamics: A demonstration that controlling gradient propagation at initialization directly shapes the NTK eigenvalues, thereby determining the learning speed of different frequency components and avoiding the spectral bias inherent in other deep network configurations.

Results
The proposed initialization ($\sigma_a = 0$) was benchmarked against the original SIREN scheme, the $\sigma_a = 1$ variant, and other state-of-the-art architectures (e.g., FINER, WIRE, Tanh with Fourier features, ReLU with positional encoding) across various tasks:

• Function Fitting: In 1D, 2D, and 3D multi-scale function approximation, the proposed method consistently achieved lower generalization error compared to baselines, particularly as network depth increased.
• Image and Video Reconstruction: In image fitting and video reconstruction tasks (using the ERA-5 atmospheric dataset), the proposed method produced sharper features and significantly fewer high-frequency artifacts compared to the original SIREN and other methods.
• Physics-Informed Neural Networks (PINNs): The method was tested on solving the 1D Burgers equation, 2D stationary Navier-Stokes equations, and the 2D heat equation in complex geometries. The proposed initialization yielded stable solutions, whereas the original SIREN initialization produced noisy, unstable results in the Navier-Stokes and heat equation tasks due to uncontrolled high-frequency components.
• Denoising: In image denoising experiments, the proposed initialization acted as an effective implicit regularizer, reconstructing clean signals while ignoring high-frequency noise, unlike other methods that tended to overfit the noise.

Significance
The paper claims that this work provides a deeper theoretical understanding of INR initialization, moving beyond empirical tuning to a principled approach based on dynamical systems theory. By stabilizing gradient propagation and controlling the frequency spectrum at initialization, the proposed method allows deep sinusoidal networks to learn high-frequency details without introducing spurious artifacts. This enables stable training dynamics with learning rates that scale linearly with depth, making SIREN architectures more robust and applicable to complex, high-dimensional problems such as scientific computing and multi-scale signal reconstruction. The authors note that while the motivation stems from INRs, the closed-form initialization is applicable to any deep architecture with periodic activations where gradient stability and frequency control are desired.