Analysing Rescaling, Discretisation, and Linearisation in RNNs for Neural System Modelling

Technical Summary: Analysing Rescaling, Discretisation, and Linearisation in RNNs for Neural System Modelling

Problem Statement
Recurrent Neural Networks (RNNs) are foundational tools in computational neuroscience for modelling neural activity, decision-making circuits, and motor control. These models typically begin as continuous-time differential equations. However, to simulate, analyse, and implement these systems computationally, researchers must apply three fundamental procedures: temporal rescaling (adjusting time scales), temporal discretisation (converting continuous equations to discrete steps for computers), and linearisation (approximating nonlinear dynamics for analytical tractability).

While each procedure is individually well-understood, the literature lacks a formal establishment of how these operations interact when applied in different sequences. Specifically, it has not been rigorously proven whether the order of application affects the resulting mathematical model. This ambiguity creates potential risks in model interpretation, particularly regarding the stability of fixed points, the preservation of controllability, and the alignment of model dynamics with experimental recordings (e.g., temporal scaling in interval production tasks).

Methodology
The authors define a standard continuous-time RNN framework governed by the differential equation:
$$h'(t) = -\lambda h(t) + \sigma(W h(t) + \tilde{W} x(t) + b)$$
where $h(t)$ is the activity vector, $x(t)$ is the input, $\sigma$ is a pointwise non-linear activation function, and $W, \tilde{W}$ are weight matrices.

The study formally defines three operators acting on this system:

1. Temporal Rescaling ($R_\tau$): A transformation $t = \tau s$ that stretches or compresses the time axis.
2. Discretisation ($D_\Delta$): The application of the forward Euler method to convert the differential equation into a discrete recurrence relation with step size $\Delta$. The authors explicitly restrict their analysis to the Euler scheme, noting that higher-order integrators (e.g., Runge-Kutta) introduce nested nonlinearities that may break commutativity.
3. Linearisation ($L$): A Taylor expansion around the origin ($\phi=0$) where $\sigma(\phi) \approx \phi$, assuming the system operates in a "regular regime" (small activity or stable decay).

The core methodology involves a pairwise commutativity analysis. The authors construct a diagrammatic framework (Figure 1) to test if applying two operations in sequence (e.g., $R_\tau$ then $D_\Delta$) yields an analytically identical equation to applying them in the reverse order ($D_\Delta$ then $R_\tau$). They define "operational equivalence" ($E_{AB} \sim E_{BA}$) as the condition where the resulting functional forms are term-by-term identical.

Key Contributions and Results
The paper provides formal proofs and numerical verification for the following findings:

1. Pairwise Commutativity: The authors prove that the three operations commute pairwise under the specified conditions:

   • $R_\tau$ and $D_\Delta$: Rescaling time and then discretising yields the same recurrence as discretising with a scaled step size and then rescaling the indices.
   • $D_\Delta$ and $L$: Linearising the continuous system and then discretising yields the exact same affine recurrence as discretising the nonlinear system and then linearising the update rule.
   • $L$ and $R_\tau$: Linearising a rescaled system is equivalent to rescaling a linearised system.

2. Structural Preservation: The commutativity guarantees that the controllability (state-space reachability) of the network is preserved across these transformations. Specifically, the algebraic independence of the operations ensures that no "blind spots" or inaccessible sub-spaces are artificially created in the discretised or rescaled models, provided the original continuous system was controllable.

3. Numerical Verification: Using a small RNN ($N=6$) with a $\tanh$ activation function and a Gaussian pulse input, the authors simulated all six possible permutations of operator pairs. The trajectories obtained from different orders overlapped exactly up to floating-point precision ($\sim 10^{-16}$), confirming the analytical proofs.

Significance and Claims
The paper claims that these results resolve operational ambiguities in computational neuroscience and machine learning workflows:

• Validation of "Black Box" Analysis: In studies where trained discrete RNNs are linearised to find fixed points and Jacobians, the commutativity of $L$ and $D_\Delta$ guarantees that the stability properties derived from the discrete model faithfully represent the underlying continuous biological system. Researchers can infer continuous attractors directly from discrete in-silico Jacobians.
• Flexible Temporal Scaling: For tasks involving temporal scaling (e.g., motor timing), the commutativity of $D_\Delta$ and $R_\tau$ allows modellers to scale the speed of a trained discrete model without retraining or re-discretising the continuous equations, saving computational resources while maintaining fidelity.
• Robustness of Linear Analysis: The commutativity of $L$ and $R_\tau$ ensures that local linear approximations (eigenvalues, eigenvectors, and topological classifications of fixed points) remain invariant under temporal rescaling, merely scaling the eigenvalues by the factor $\tau$.

Limitations and Scope
The authors explicitly state that these exact commutativity results are properties of the Euler discretisation framework. They do not claim these results hold for higher-order integrators (like Runge-Kutta) or adaptive-step solvers, as these methods introduce higher-order interaction terms that generally prevent exact commutativity. Furthermore, the linearisation is valid primarily in the "regular regime" near the origin or for systems with asymptotic stability; the paper acknowledges that highly nonlinear phenomena (e.g., chaos or bifurcations far from equilibrium) may not be captured by these linear approximations.

The work serves as a mathematical foundation for the flexible design of biologically plausible RNNs, ensuring that the structural properties of neural dynamics are preserved regardless of the specific order in which researchers apply rescaling, discretisation, and linearisation.