Stabilization of recurrent neural networks through divisive normalization

1. Problem Statement

Recurrent Neural Networks (RNNs) are fundamental to both biological cognition and engineered systems, enabling complex computations through feedback loops. However, a major theoretical and practical challenge is stability.

• The Instability Paradox: In standard linear dynamical models, stability requires that all eigenvalues of the recurrent weight matrix lie within the unit circle (spectral radius $\rho < 1$). If recurrent weights are strong enough to support complex computations (often requiring $\rho > 1$), the network typically becomes unstable, leading to "exploding" dynamics.
• Biological Context: Biological neural circuits exhibit strong recurrent connections yet remain stable. The paper investigates how divisive normalization—a canonical neural computation where a neuron's response is divided by the summed activity of a population—contributes to this stability.
• The Gap: While normalization is known to explain sensory phenomena (e.g., contrast saturation), its role in stabilizing high-dimensional, randomly connected recurrent networks with arbitrary weights was not previously established.

2. Methodology

The authors employed a combination of numerical simulations and analytical perturbation theory using the ORGaNICs (Oscillatory Recurrent Gated Neural Integrator Circuits) model.

• The Model (ORGaNICs):
  • The model consists of principal excitatory neurons ($y$) and a secondary inhibitory population ($a$) that acts as a multiplicative gain modulator.
  • The dynamics are governed by coupled differential equations where the firing rate of the excitatory neuron is squared ($y^+ = y^2$) and the inhibitory neuron follows a square-root compression ($a^+ = \sqrt{a}$).
  • Crucially, the recurrent term is multiplied by a gain factor $(1 - a^+)$, implementing divisive normalization dynamically.
• Network Architecture:
  • The authors extended the model from a simple 2-neuron circuit to a high-dimensional network ($N$ neurons) with arbitrary recurrent connections.
  • The recurrent weight matrix $W$ is constructed as $W = I + K$, where $I$ is the identity and $K$ is a symmetric random matrix drawn from the Gaussian Orthogonal Ensemble (GOE). This ensures balanced excitation and inhibition (mean zero) with a controllable spectral radius $\rho(W) = 1 + 2\Delta$.
• Analysis Techniques:
  • Numerical: Simulations using the explicit Euler method to find fixed points and compute the Jacobian matrix eigenvalues ($\lambda$) to determine stability.
  • Analytical: Perturbation theory was used to expand the fixed-point solution around the identity matrix. This allowed the derivation of the mean and variance of neural responses as a function of the recurrent strength $\Delta$ and input drive $z$.
  • Stability Metrics: Stability was assessed by the real part of the largest eigenvalue of the Jacobian ($\lambda_{max}$).
    • $\lambda_{max} < 0$: Stable.
    • $\lambda_{max} \to 0$: Critical Slowing Down (CSD) (early warning of instability).
    • $\lambda_{max} > 0$: Unstable (divergence or limit cycles).

3. Key Contributions

1. Stabilization via Normalization: The paper demonstrates that ORGaNICs can remain stable even when the spectral radius of the recurrent matrix exceeds 1 ($\rho > 1$), a regime where standard linear models would immediately diverge. The multiplicative normalization mechanism dynamically scales the effective recurrent strength, keeping the system stable.
2. Identification of Critical Slowing Down (CSD): The authors identify a specific regime preceding instability where the network exhibits critical slowing down. In this phase, the network converges to its fixed point extremely slowly (time constants $\gg$ intrinsic neuron time constants).
3. Theoretical Link between Normalization and Stability: A major theoretical contribution is the proof that the breakdown of divisive normalization coincides exactly with the onset of critical slowing down.
   • Normalization is defined as the state where the mean response exceeds the standard deviation across network realizations ($E[y] > \sqrt{Var[y]}$).
   • The authors derived an analytical threshold $\Delta_{loss}(z)$ where normalization fails. They proved that $\Delta_{loss}(z) \approx \Delta_{csd}(z)$, meaning the loss of normalization is a precise predictor of the onset of dynamical instability.
4. Early Warning Signal: The study proposes that the loss of normalization (detectable via increased neural variability) serves as an early warning signal for pathological states like seizures, occurring before the network actually becomes unstable.

4. Key Results

• Phase Diagrams: The authors mapped the stability landscape in the $(z, \Delta)$ plane (Input Drive vs. Recurrent Strength).
  • Stable Phase: Low $\Delta$, fast convergence.
  • Critically Slowed Phase: Intermediate $\Delta$. The system is stable but converges slowly. Here, the largest eigenvalue $\lambda \approx 0$.
  • Unstable Phase: High $\Delta$. The system enters limit cycles or diverges.
• Quantitative Agreement: The analytical prediction for the loss of normalization threshold (Eq. 8) matched numerical simulations with high precision across various input drives and network sizes ($N=100$ to $1000$).
• Spectral Properties: In the critically slowed regime, the Jacobian spectrum contains a large number of eigenvalues approaching zero, indicating the emergence of multiple long time scales without fine-tuning of weights.
• Robustness: The findings hold for both delocalized (uniform) and localized (sparse) input drives. The results also extend to networks with Excitation-Inhibition (E-I) imbalances, though the specific thresholds shift.
• Oscillations: The transition to instability is often preceded by damped oscillations (spiraling attractors) in the gamma frequency range (30–100 Hz), which become more pronounced as normalization breaks down.

5. Significance

• Biological Plausibility: The results suggest that the widespread implementation of divisive normalization in biological systems (visual, olfactory, auditory, and cognitive circuits) is not merely for computational efficiency (e.g., contrast invariance) but is essential for dynamical stability. It allows the brain to utilize strong recurrent connections necessary for complex processing without succumbing to instability.
• Clinical Implications: The link between normalization breakdown and critical slowing down offers a new biomarker for neurological disorders.
  • Epilepsy: Seizures are often preceded by critical slowing down. The paper suggests that a failure of normalization mechanisms could be the underlying cause.
  • Neurodevelopmental Disorders: The findings align with observations in autism, schizophrenia, and depression, where reduced normalization and increased neural variability are reported.
• Engineering Applications: For artificial neural networks, incorporating divisive normalization (or similar multiplicative gating) could allow for the training of deeper, more recurrent architectures with stronger weights, improving performance while maintaining stability.
• Theoretical Framework: The work bridges random matrix theory, dynamical systems, and neuroscience, providing a rigorous analytical framework to predict tipping points in complex neural systems.

In summary, Morone et al. provide a compelling argument that divisive normalization is a stabilizing mechanism that permits strong recurrent interactions. The breakdown of this normalization serves as a mathematically predictable and experimentally detectable early warning signal for the loss of stability in neural circuits.