R2DN: Scalable Parameterization of Contracting and Lipschitz Recurrent Deep Networks

Technical Summary: R2DN: Scalable Parameterization of Contracting and Lipschitz Recurrent Deep Networks

Problem Statement
Deep neural networks (DNNs), particularly Recurrent Neural Networks (RNNs), have achieved significant success in modeling dynamical systems. However, standard DNNs are often sensitive to input perturbations, leading to brittle behavior and instability in control applications. While architectures like the Recurrent Equilibrium Network (REN) have been proposed to guarantee internal stability (contraction) and input-output robustness (Lipschitz continuity) via direct parameterization, they suffer from a critical computational bottleneck. RENs require the iterative solution of an implicit equilibrium layer at every time step during inference and backpropagation. This process is computationally expensive, hinders scalability on GPU architectures designed for parallelism, and limits the feasible model size, batch size, and sequence length.

Methodology
The authors propose the Robust Recurrent Deep Network (R2DN), a scalable parameterization that retains the stability and robustness guarantees of RENs while eliminating the computational overhead of implicit layers. The R2DN is constructed as the feedback interconnection of a Linear Time-Invariant (LTI) system and a 1-Lipschitz deep feedforward network (DNN).

The core architectural modification involves two key changes to the REN structure:

1. Elimination of the Equilibrium Layer: The feedthrough term ($D_{11}$) in the LTI system is set to zero. This removes the implicit equation $w_t = \sigma(D_{11}w_t + \dots)$, allowing the network state to be computed explicitly rather than iteratively.
2. Scalable Nonlinearity: The scalar activation function $\sigma$ used in RENs is replaced by a general 1-Lipschitz deep feedforward network $\phi_g$ (e.g., MLPs, CNNs, ResNets).

The paper provides a direct parameterization for these networks, mapping learnable parameters $\theta$ directly to weights and biases such that the resulting model is guaranteed to be:

• Contracting: Ensuring internal states forget initial conditions exponentially, regardless of the specific equilibrium point.
• Lipschitz Robust: Satisfying incremental Integral Quadratic Constraints (IQCs) to bound output sensitivity to input perturbations.

The parameterization relies on constructing the LTI system matrices ($A, B, C, D$) to satisfy specific Linear Matrix Inequalities (LMIs) derived from the small-gain theorem and IQC theory. Specifically, the authors derive conditions where the LTI system admits an incremental IQC defined by $(Q, S, R)$, ensuring the overall interconnection with the 1-Lipschitz DNN remains stable and robust.

Key Contributions

• Architectural Innovation: The introduction of R2DN, which replaces the implicit equilibrium layer of RENs with an explicit structure, enabling direct computation without iterative solvers.
• Scalable Parameterization: A method to parameterize contracting and Lipschitz R2DNs that supports arbitrary 1-Lipschitz DNN architectures, unlike RENs which are restricted to specific scalar activation structures or lower-triangular feedthroughs.
• Computational Efficiency: The removal of the equilibrium layer allows R2DNs to leverage massive parallelism on GPUs, significantly speeding up both inference and backpropagation compared to RENs.
• Theoretical Guarantees: Proof that the proposed parameterization ensures contraction and incremental IQC satisfaction by construction, without requiring constrained optimization solvers.

Results
The authors evaluated R2DNs against RENs on three representative tasks: nonlinear system identification (F16 ground vibration data), nonlinear observer design for a reaction-diffusion PDE, and learning-based feedback control.

• Speed: Training and inference for R2DNs were found to be up to an order of magnitude faster than RENs. The speedup is attributed to the elimination of iterative equilibrium solving, which is particularly beneficial for long-horizon simulations and large batch sizes.
• Performance: Despite the architectural simplification and speed gains, R2DNs achieved similar test set performance (loss and error metrics) to RENs across all three tasks.
• Scalability: Experiments on model expressivity showed that while computation time for RENs increases rapidly with model complexity, R2DNs scale much more favorably, maintaining low inference times even as model expressivity increases.

Significance and Claims
The paper claims that R2DNs offer a practical, computationally efficient alternative to RENs for robust recurrent modeling in machine learning and data-driven control. By removing the requirement to solve implicit equations, R2DNs make it feasible to scale up network size, batch size, and input sequence length, which was previously prohibitive with RENs. The authors emphasize that this approach retains the strong theoretical guarantees of stability and robustness while enabling the use of modern, high-capacity DNN architectures (like Transformers or ResNets) within a recurrent framework. The work suggests that R2DNs are well-suited for applications requiring frequent model inference on small input batches, such as real-time control, where the computational overhead of RENs is a limiting factor.

Future work identified by the authors includes extending the parameterization to allow for non-zero $D_{22}$ (direct feedthrough from input to output) in Lipschitz R2DNs and further studying the scalability of these models in higher-dimensional machine learning tasks.