🧬 biology

Divisive Normalization Shapes Low-Rank Slow Manifolds for Continuous Working Memory

This paper proposes the Recurrent Divisive Normalization Network (RDNN), a biologically inspired model that utilizes divisive normalization to overcome the fragility of classical attractor networks and the discretization failures of standard RNNs, thereby enabling the robust learning of continuous, low-rank slow manifolds for working memory through activity-dependent gradient scaling.

Original authors: Zhaotian Gu, Jie Su, Weiwei Wang, Chang Liu, Tianyi Qian, Dahui Wang

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhaotian Gu, Jie Su, Weiwei Wang, Chang Liu, Tianyi Qian, Dahui Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine your brain as a bustling city where thoughts are like cars driving on roads. Sometimes, you need to keep a car moving smoothly along a specific route without stopping, even if the traffic lights change or the wind blows. This ability to hold a continuous thought—like remembering a direction you're facing or a number you're counting—is called "working memory." For decades, scientists have tried to build computer models that mimic this skill. The old models were like rigid train tracks: if you tried to make them carry a continuous variable, they would snap into a series of disconnected stops, forcing the train to jump from one station to the next instead of gliding smoothly. Other models were like delicate glass sculptures; they could hold a continuous path, but the slightest wobble would shatter the whole thing. The big question has been: how does the real brain manage to keep these thoughts flowing smoothly and stably, and can we build a computer that does the same without breaking?

This paper introduces a new kind of computer model called the Recurrent Divisive Normalization Network (RDNN) to solve this puzzle. The authors took inspiration from a biological rule found in real brains called "divisive normalization." Think of this as a smart volume knob that doesn't just turn sound up or down, but adjusts the volume based on how loud the whole room is. If everyone in the room starts shouting, the knob automatically turns down the gain for everyone so no one drowns out the others, keeping the conversation balanced. The researchers built a digital version of this "volume knob" into their network. They found that this simple mechanism acts like a magical stabilizer. Instead of shattering the smooth path of memory into jagged, disconnected points, the RDNN keeps the memory flowing on a "slow manifold"—a smooth, continuous highway where thoughts can travel without getting stuck or drifting away.

The team tested their model on tasks that require holding a continuous variable, like tracking a spinning wheel or remembering a direction without looking at it. While standard computer models (like GRUs and LSTMs) failed to learn these smooth paths, often breaking the memory into a series of frozen snapshots, the RDNN succeeded. It learned to maintain a stable, continuous representation that could handle changing inputs without collapsing. The researchers discovered that this "volume knob" does something surprising behind the scenes: it naturally forces the network to compress its own complexity. Even though the network has many parts, the math shows it only really uses a few key dimensions to do the work, making it efficient and robust.

Crucially, the paper argues that this specific type of "volume control" (divisive) is mathematically essential for handling changing inputs. The authors tested a different kind of control, called "subtractive inhibition," which is like simply subtracting a fixed amount of noise. They found that while subtractive control could hold a static memory (like remembering a direction when nothing is happening), it failed miserably when the input started moving. The smooth path shattered into a messy collection of stops. This suggests that for the brain to handle the dynamic, changing world, it needs the multiplicative, scaling power of divisive normalization, not just simple subtraction.

In simulations, the RDNN showed that it could learn these smooth paths with high fidelity, maintaining a low "effective rank" (a measure of how many dimensions the network actually uses) that stayed small even as the network got bigger. The authors suggest that this mechanism isn't just a biological accident but a critical computational trick that allows for high-fidelity, continuous working memory. However, they also note that while this model is great for short-term, dynamic tracking, it might eventually drift over very long periods, whereas the "shattered" models might actually be better at locking in a memory for a very long time. The paper concludes that by mimicking this specific biological rule, we can build artificial brains that are much better at the fluid, continuous thinking that humans do every day.

Technical Summary: Divisive Normalization Shapes Low-Rank Slow Manifolds for Continuous Working Memory

1. Problem Statement

The ability to robustly maintain and update continuous variables (e.g., spatial location, head direction) is a hallmark of biological working memory. However, existing computational models face a fundamental dichotomy:

Continuous Attractor Networks (CANs): While theoretically capable of storing graded values, they suffer from the "fine-tuning problem," where infinitesimal perturbations in recurrent dynamics destroy the continuous manifold, causing memory to drift or collapse.
Standard Artificial RNNs (GRUs, LSTMs): Despite success in sequential tasks, dynamical systems analyses reveal they struggle to learn true continuous attractors. Instead, they tend to discretize the continuous state space, shattering the manifold into localized point attractors separated by saddles.

The core question addressed is: What computational mechanisms allow biological networks to robustly represent continuous working memory, and how can these be integrated into learnable models to avoid the topological discretization of standard architectures?

2. Methodology

The authors propose the Recurrent Divisive Normalization Network (RDNN), a minimal, algebraically isolated model inspired by divisive normalization (DN), a canonical neural computation observed in cortical circuits for dynamic gain control.

Model Architecture

The RDNN consists of two interacting populations governed by coupled stochastic differential equations (SDEs):

Principal Excitatory Population ( $R$ ): Represents the working memory state.
Auxiliary Inhibitory Pool ( $G$ ): Provides dynamic gain control.

The continuous-time dynamics are defined as:
$\tau_R dR = \left( -R + \frac{Jf(R) + I(t)}{\eta + G} \right) dt + \sigma_R dW_R$
$\tau_G dG = (-G + Wf(R)) dt + \sigma_G dW_G$

Key features include:

Divisive Bottleneck: The inhibitory state $G$ (tracking total activity via $Wf(R)$) divides the excitatory drive, implementing shunting inhibition.
Biophysical Constraints: Parameters are constrained to be positive to adhere to Dale's law, ensuring a positive dynamical system.
Training: The model is trained using Backpropagation Through Time (BPTT) on two canonical tasks:
1. Angular Velocity Integration: Input-driven updating of a continuous variable.
2. Memory-guided Saccade: Autonomous maintenance of a static memory over a delay period.

Analytical Framework

The study employs dynamical systems analysis to characterize learned representations, focusing on:

Topological Distribution: Classification of fixed points (stable, unstable, saddle, marginal) and limit cycles.
Spectral Properties: Analysis of Jacobian eigenvalue spectra to assess time-scale separation and normal hyperbolicity.
Effective Rank: Quantification of the intrinsic dimensionality of the recurrent connectivity using the 99% energy effective rank.
Ablation: Comparison with a Subtractive Network (replacing division with additive inhibition) and explicit low-rank factorization variants.

3. Key Contributions

A. Biologically-Inspired Architecture for Continuous Memory

The RDNN demonstrates that incorporating divisive normalization allows networks to learn robust slow manifolds for continuous working memory. Unlike standard GRUs and LSTMs, which fail to converge on the memory-guided saccade task or discretize the state space, the RDNN successfully maintains continuous representations.

B. Emergent Low-Rank Dynamics via Gradient Scaling

The paper identifies a mechanism where divisive normalization induces self-compression of the network's effective rank without explicit low-rank parameterization.

Activity-Dependent Gradient Attenuation: During BPTT, the dynamic divisor ( $\eta + G$ ) scales gradients inversely to network activity. In highly active regimes, gradient updates are dampened.
Consequence: This local scaling restricts major parameter updates to a subset of states, empirically forcing the network into a tight, low-dimensional subspace.
Optimization Anomaly: The authors observe that explicitly factorizing weights (forcing a low-rank bottleneck) introduces a "higher-rank convergence anomaly," where wider bottlenecks hinder optimization due to non-convex scaling symmetries. The unconstrained RDNN avoids this pathology while naturally achieving low-rank dynamics.

C. Mechanistic Distinction: Divisive vs. Subtractive Inhibition

Through systematic ablation, the paper delineates the functional niches of inhibition types:

Subtractive Inhibition: Sufficient for static memory maintenance (autonomous regime) where inputs are zero. It can form marginal fixed points but fails under time-varying inputs.
Divisive Normalization: Essential for input-driven integration. It prevents "manifold shattering" under external drive by providing contrast invariance and dynamic gain control. Without it, the continuous manifold breaks into discrete point attractors.

4. Key Results

Dynamical Topology

RDNN: Exhibits a fluid topological structure dominated by saddle points (acting as slow channels) and marginal fixed points (approximating continuous attractors). In the autonomous saccade task, the state space is overwhelmingly dominated by marginal fixed points with a strict spectral gap.
Baselines (GRU/LSTM): Form alternating discrete stable fixed points and saddles, effectively discretizing the continuous variable. They fail to maintain continuous attractors in the memory-guided saccade task.

Spectral Properties

The RDNN displays a structured tripartite eigenvalue spectrum: a cluster near 0 (fast dissipation), off-axis conjugate pairs (transient stabilization), and boundary conjugate pairs near the unit circle (marginally stable rotational dynamics).
This structure ensures normal hyperbolicity, allowing the network to separate fast and slow time scales effectively.

Effective Rank and Compression

Sub-linear Scaling: While GRU/LSTM effective ranks scale linearly with hidden size $H$ (exploiting all degrees of freedom), the RDNN exhibits highly sub-linear, flat scaling (e.g., at $H=256$ , RDNN rank $\approx 35$ , while baselines $\approx 240$ ).
Hyper-Compression: Even when explicitly forced into a low-rank bottleneck ( $r=32$ ), the RDNN compresses its effective rank further to $\approx 2\text{--}4$ , indicating the intrinsic dimensionality of the task is extremely low.

Ablation Findings

In the Angular Velocity Integration task, the Subtractive Network's manifold shatters into a discretized space populated by stable fixed points, leading to higher effective ranks and redundant slow pathways.
In the Memory-guided Saccade task (zero input), both networks perform similarly, confirming that divisive normalization is specifically critical for handling time-varying inputs and preventing attractor shattering.

5. Significance and Claims

The paper claims that divisive normalization is not merely a biological artifact for sensory gain control, but a critical computational mechanism for learning high-fidelity continuous representations.

Theoretical Insight: It provides a unified framework explaining how biological gain control shapes the topological landscape of recurrent memory circuits, overcoming the fine-tuning fragility of classical CANs and the discretization bias of standard RNNs.
Optimization Perspective: The study suggests that biologically inspired multiplicative gain control offers an optimization-friendly route to discovering low-dimensional cognitive representations, avoiding the non-convex anomalies associated with explicit low-rank factorization.
Functional Synergy: The findings suggest a functional synergy in cortical circuits where divisive normalization handles contrast-invariant dynamic tracking, while subtractive mechanisms may provide noise-resistant stability over long timescales (via error-correcting "lock-in" effects).

The authors maintain modesty regarding limitations, noting that the current model relies on BPTT (lacking biological plausibility for synaptic plasticity) and focuses exclusively on continuous memory, leaving mixed discrete-continuous representations and multi-component inhibitory synergies for future study.