Structure Over Nonlinearity: Explicit Interaction Architectures for Dynamical Learning

Technical Summary: Structure Over Nonlinearity: Explicit Interaction Architectures for Dynamical Learning

Problem Statement
Current learning architectures for dynamical systems predominantly rely on generic nonlinear function approximators (e.g., recurrent neural networks) to capture complex temporal behaviors. While flexible, these "black-box" approaches often require vast numbers of parameters and extensive training, offering limited insight into the underlying system structure. Their expressive power is derived largely from the capacity of nonlinear mappings rather than the organization of the model itself. Consequently, they may struggle to efficiently capture structured dynamical phenomena, particularly when data is limited or interpretability is required. Existing physics-inspired approaches (e.g., neural differential equations) attempt to introduce inductive bias but often retain reliance on highly expressive nonlinear components or suffer from computational issues related to implicit solvers.

Methodology
The authors propose an alternative paradigm where modeling capability arises primarily from structure rather than generic nonlinearities. The core methodology involves the design of Explicit Structured Dynamical Units based on wave-inspired interaction structures.

• Unit Architecture: The proposed unit operates on an input $a_n$ and internal state $\xi_n$. It processes data through a sequence of local transformations ($R_1, R_2, R_3$) modulated by the internal state, followed by a nonlinear activation $\sigma$. Crucially, the unit adopts a strictly causal, layered organization that eliminates algebraic loops. This ensures all computations are explicit and can be scheduled sequentially without requiring iterative fixed-point solvers or implicit equation solutions.
• State Evolution: The internal state updates via $\xi_{n+1} = \alpha \odot \xi_n + \phi(b_n, \xi_n)$, where $\phi$ is a nonlinear update function. This formulation yields a fully explicit dynamical system.
• Layered Composition: Multiple units are stacked to form layered architectures. Unlike standard deep networks where depth increases static representational capacity, stacking these units induces hierarchical dynamical behavior. Each layer maintains its own independent internal state, allowing the model to capture interacting dynamical processes at different time scales. The first layer captures input-driven local transformations, while subsequent layers refine and integrate these representations.
• Theoretical Properties: The paper establishes that these units are explicitly computable at each time step. It also provides a sufficient condition for BIBO (Bounded-Input Bounded-Output) stability based on the Lipschitz continuity of activation functions and contractivity of the state update, linking the architecture to passivity-based stability analysis found in wave digital structures.

Key Contributions

1. Structured Dynamical Unit: Introduction of a computational unit based on explicit, wave-inspired interaction structures with internal state, designed to eliminate algebraic loops and ensure explicit computability.
2. Layered Hierarchical Architecture: A method for composing these units into layered models that enable hierarchical dynamical modeling, where depth introduces additional dynamical degrees of freedom rather than just static mapping capacity.
3. Experimental Evidence: Demonstration that structure and depth improve generalization in nonlinear system identification tasks, even under conditions of minimal parameter optimization.

Experimental Results
The authors evaluated the proposed framework on a nonlinear system identification task involving input signals with smooth stochastic components, impulsive events, and oscillatory behavior. They compared one-layer and two-layer models under three training configurations: fixed parameters, readout-only least-squares fitting, and partial optimization of final stage parameters.

• Generalization: The two-layer model consistently outperformed the one-layer model on both training and validation data. The improvement was more pronounced on the validation set, suggesting that depth enhances generalization rather than merely increasing fitting capacity.
• Role of Structure: Under "readout-only" fitting (where internal parameters remain fixed and only the output layer is trained), the two-layer architecture achieved significantly lower validation error (0.004 MSE) compared to the one-layer model (0.009 MSE).
• Minimal Optimization: The results indicated that the performance gains were largely induced by the fixed structured dynamics and the fitted linear readout, with partial optimization providing no appreciable additional improvement at the reported precision. This suggests that useful dynamical coordinates emerge from the organization of interactions prior to substantial parameter adaptation.

Significance and Claims
The paper posits that interaction structure serves as a primary source of model expressivity in dynamical learning systems, offering a viable alternative to conventional black-box approaches.

• Structure-First Design: The findings support the view that carefully designed dynamical architectures with built-in inductive bias can achieve strong performance without relying on increasingly expressive nonlinear models.
• Emergent Representations: The study demonstrates that informative internal representations can emerge from the organization of local interactions even before significant parameter optimization, challenging the notion that learning is solely about discovering arbitrary input-output mappings.
• Scalability and Interpretability: By eliminating algebraic loops, the proposed explicit units enable scalable layered architectures compatible with learning frameworks, bridging the gap between data-driven learning and physically motivated modeling (e.g., wave digital structures).

The authors characterize this work as a "first learnable instantiation" of a broader class of Wave Digital Computational Structures (WDCS). They modestly frame the results as preliminary evidence that structure-aware computational systems constitute a viable research direction, rather than claiming immediate state-of-the-art performance on standard benchmarks. Future work is identified as necessary to develop complete theoretical characterizations, advanced training strategies, and the incorporation of physical constraints like passivity and energy conservation.