Technical Summary: Scaling Laws and Tradeoffs in Recurrent Networks of Expressive Neurons

Problem Statement

Cortical neurons are complex, multi-timescale processors capable of nonlinear integration beyond simple point-wise nonlinearities. In contrast, mainstream machine learning (ML) predominantly relies on extremely simple units (e.g., standard perceptrons or LSTMs), a default inherited from early neural network theory. While computational neuroscience has refined single-neuron models to include rich internal dynamics and structured dendritic integration, and ML has scaled performance by widening networks and stacking layers, the two fields have largely evolved along separate axes.

The central normative question addressed is: How should a fixed parameter budget $P$ be optimally allocated between the number of units $N$, the per-unit effective complexity $k_e$, and the per-unit connectivity $k_c$? Current ML scaling laws focus on model size, data, and compute, but scaling with per-unit complexity remains largely unexplored. The authors posit that the assumption that simple units are an obviously optimal default may be flawed once the tradeoff surface between unit count, unit complexity, and connectivity is probed.

Methodology

The ELM Network Architecture

To investigate this tradeoff, the authors introduce the ELM Network, a recurrent architecture built from Expressive Leaky Memory (ELM) neurons. This design allows for the independent tuning of $N$, $k_e$, and $k_c$ while maintaining stable training across orders of magnitude.

• The ELM Neuron: A recurrent cell featuring:
  • Structured Synaptic Integration: Inputs are weighted, grouped into branches, and summed.
  • Nonlinear Internal Computation: Branch activations and decayed memory states pass through a multi-layer perceptron (MLP).
  • Multi-Timescale Memory: Leaky integration of memory units with learnable timescales.
  • Temporal High-Pass Filtering: An exponential moving average (EMA) of the linear memory readout is subtracted before a ReLU activation, stabilizing training and filtering slow oscillations.
• The Network: Consists of a wide recurrent hidden layer of ELM neurons, followed by a simpler feed-forward readout layer and a linear output projection.
• Budget Definition: The total parameter budget is defined as $P = N(k_e + k_c)$, where $k_e$ represents internal neuron parameters (MLP weights, readout weights) and $k_c$ represents connection parameters (synaptic weights).

Experimental Setup

The authors evaluated the model on two qualitatively different benchmarks:

1. SHD-Adding: A neuromorphic benchmark involving the summation of spoken digits encoded as spatio-temporal spike patterns.
2. Enwik8: A character-level language modeling task using the first $10^8$ bytes of Wikipedia.

Experiments involved sweeping $N$, $k_e$ (via memory units $d_m$ and MLP depth/width), and connectivity ($k_c$) across three orders of magnitude in trainable parameters.

Theoretical Framework

To explain the empirical tradeoffs, the authors propose a closed-form information-theoretic model treating the neural layer as a collection of noisy channels.

• Effective Representation Information ($I_{rep}$): Defined as the mutual information between the task-relevant signal and the noisy layer output.
• Key Assumptions:
  1. Gaussian signal and noise.
  2. Power-law signal covariance spectrum ($\lambda_i \propto i^{-\beta}$), where $\beta$ captures inter-neuron redundancy.
  3. Per-neuron noise decays as a power law with complexity $k_e$ until saturating at a floor ($\sigma_n^2 \propto \max((\gamma k_e)^{-\alpha}, q_\infty)$), where $\alpha$ captures spatio-temporal integration expressivity.
  4. Fixed budget $P$ distributed among $N$ neurons.
• Resulting Formula:
  $$I_{rep}(k_e) = \frac{1}{2} \sum_{i=1}^{P/(k_e+k_c)} \log_2 \left( 1 + s(k_e) i^{-\beta} \right)$$
  where $s(k_e)$ is the signal-to-noise ratio dependent on $k_e$.

Key Results

1. Monotonic Scaling vs. Non-Trivial Tradeoffs

• Individual Axes: Performance improves monotonically when scaling $N$, $k_e$, or connectivity ($k_c$) individually.
• Fixed Budget Tradeoff: Under a fixed parameter budget, a clear, non-trivial optimum emerges in the tradeoff between $N$ and $k_e$.
  • Networks with too few neurons or too simple neurons perform poorly.
  • The optimum is sharp, particularly when using binary spiking surrogates.
• Budget Shift: As the total budget $P$ increases, the optimal configuration shifts toward both more neurons and more complex neurons (e.g., shifting from $N=512, d_m=15$ to $N=1024, d_m=25$ when the budget quadruples).

2. Connectivity and Recurrence

• Connectivity: Increasing the number of synapses ($d_s$) improves performance. Once sufficient feedforward connections exist, additional performance gains are driven almost entirely by recurrent connections.
• Recurrence Fraction: Larger networks favor a higher fraction of recurrent connections ($\rho_{rec}$).
• Complexity vs. Connectivity: Splitting parameters between neuron complexity and connectivity yields an optimum around $N_{rec}$ connections.

3. Theoretical Validation

• The information-theoretic model qualitatively reproduces the empirical scaling shapes, including the location and shift of the optimum with budget.
• Parameter Interpretability:
  • $\alpha$ (noise decay exponent) correlates with spatio-temporal integration settings (e.g., $l_{mlp}$, $\tau_{max}$). Empirical measurements on single-neuron benchmarks confirm a power-law reduction in error with $k_e$.
  • $\beta$ (spectral decay exponent) correlates with connectivity. Reduced connectivity increases $\beta$ (higher redundancy), shifting the optimum toward larger neurons.
• A joint fit of the theory to experimental data across seven ablation experiments achieved a Pearson correlation of $r=0.98$, with fitted parameters matching independent measurements.

4. Scaling Recipe

A structured hyperparameter search identified a simple scaling recipe that traces the empirical Pareto frontier across both datasets:

• $d_m \sim \sqrt{N_{rec}}$
• $d_{mlp} = 2d_m$
• $d_{tree} = 2d_{mlp}$
• $d_{branch} = d_{tree}$
  This recipe scales per-neuron connectivity aggressively, consistent with the theoretical importance of $\beta$.

Significance and Claims

The paper claims to provide a normative lens on the reliance of the cortex on complex spatio-temporal integrators. By demonstrating that simple-unit defaults are not obviously optimal under budget constraints, the work suggests that:

1. Complex per-unit machinery is a budget-efficient design choice for recurrent computation.
2. The tradeoff between unit count and unit complexity is governed by fundamental information-theoretic principles: per-neuron signal-to-noise saturation and across-neuron redundancy.
3. The "simple-unit" default in ML may be suboptimal once the full tradeoff surface is explored.

The authors explicitly note limitations: the framework abstracts away specific network topologies and optimization dynamics; assumptions (like power-law spectra) hold approximately; and the use of parameter count as a budget proxy does not directly capture metabolic costs, limiting direct biological interpretability. Nevertheless, the work establishes a robust model system for studying scaling in expressive neurons and offers a compact theoretical framework for understanding architectural tradeoffs.