Time-Varying Deep State Space Models for Sequences with Switching Dynamics

Technical Summary: Time-Varying Deep State Space Models for Sequences with Switching Dynamics

Problem Statement
The identification and modeling of time-varying systems remain a fundamental challenge in signal processing and system identification. Many real-world processes, such as financial markets, weather, and neural activity, exhibit inherent time-varying dynamics where underlying system behaviors evolve over time. While State Space Models (SSMs) offer a principled framework for system identification, classical Linear Dynamical Systems (LDS) assume fixed, time-invariant transition dynamics. Extensions like Switching Linear Dynamical Systems (SLDS) handle regime changes but require explicit estimation of the number of modes and model orders, making learning challenging. Conversely, modern Deep SSMs have achieved high performance on long sequences but traditionally rely on time-invariant parameters or input-dependent mechanisms, which may not explicitly capture smooth, continuously time-varying dynamics independent of the input.

Methodology
The authors propose a novel class of Time-Varying Deep State Space Models (TV-SSM). This framework integrates time-varying basis function expansions into deep SSM architectures.

• Model Formulation: The core innovation replaces the fixed matrices $A, B, C$ of a standard SSM with time-dependent matrices $A[t], B[t], C[t]$. Each element of these matrices is modeled as a linear combination of learnable basis functions:
  $$[A[t]]_{ij} = \sum_{k=1}^{K_A} a^{(k)}_{i,j} \times \phi^{(k)}_{A,i,j}[t]$$
  Similar expansions apply to $B[t]$ and $C[t]$. The basis functions (e.g., Gaussian functions, sine functions, or constants) are fixed, while their coefficients are learnable parameters. This allows the model to represent smooth, continuous dynamics without explicitly defining switching modes.
• Stability: To ensure the stability of the time-varying dynamics, the authors enforce a constraint on the diagonal elements of $A[t]$ (assuming a diagonal structure for simplicity). During training, if the sum of the absolute values of the coefficients for a diagonal element exceeds 1, a scaling strategy is applied to re-normalize the coefficients, ensuring the eigenvalues remain strictly within the unit circle.
• Complexity: While the number of trainable parameters increases by a factor of the dictionary size ($K_A, K_B, K_C$) per matrix element, the inference complexity remains comparable to time-invariant models. The time-dependent matrices can be pre-computed during the inference phase, resulting in the same Multiply–Accumulate (MAC) operations as a standard SSM. However, space complexity increases due to the storage of time-dependent parameters.

Key Contributions

1. Novel Framework: The paper introduces a deep SSM framework that parameterizes state transition, input, and output matrices over time using learnable basis functions, enabling the modeling of smooth, continuously time-varying dynamics without explicit mode switching.
2. Experimental Validation: The approach is validated on two distinct tasks:
   • Synthetic Data: A four-mode Switching Linear Dynamical System (SLDS) where the ground truth involves switching dynamics.
   • Real-World Data: A speech denoising task where clean speech is corrupted by noise generated from a four-mode SLDS, simulating recurring noise patterns like rotating machinery.
3. Architectural Analysis: The authors investigate trade-offs regarding the allocation of basis functions across the $A, B,$ and $C$ matrices, the impact of model depth, and the relationship between parameter counts and performance.

Results

• Synthetic Switching Systems: On the four-mode system, the fully time-varying model consistently outperformed time-invariant counterparts. Notably, the study revealed that making the state transition matrix $A[t]$ time-varying alone was insufficient to capture switching dynamics in $B$ or $C$; conversely, making $B[t]$ or $C[t]$ time-varying could compensate for fixed $A$. The time-invariant models tended to learn an "average" across modes, resulting in higher Mean-Square Error (MSE).
• Speech Denoising: In the audio denoising task, the time-varying models significantly outperformed time-invariant models.
  • Performance: The best time-varying configuration achieved a Signal-to-Noise Ratio (SI-SNR) of 19.6 dB (starting from 5 dB noise), whereas time-invariant models plateaued around 7.8 dB, regardless of network depth or activation functions.
  • Parameter Efficiency: Even when the total number of trainable parameters was matched between time-varying and time-invariant models, the time-varying approach yielded superior results. Increasing the number of parameters per neuron improved the time-varying model's performance but had no effect on the time-invariant model, suggesting the latter cannot leverage additional capacity to capture non-stationary dynamics.
  • Basis Allocation: Allocating the basis function budget entirely to the output matrix $C$ or input matrix $B$ yielded better performance than allocating it to the state transition matrix $A$.

Significance and Claims
The paper claims that the proposed time-varying model offers a flexible and efficient solution for system identification tasks involving switching or non-stationary dynamics. By modeling time-dependent dynamics explicitly through basis functions, the approach avoids the complexity of estimating discrete switching modes required by SLDS while outperforming standard time-invariant Deep SSMs. The authors note that the model maintains comparable computational complexity during inference to its time-invariant counterparts.

The study concludes that while larger time-invariant models cannot compensate for the lack of inherent flexibility in capturing switching dynamics, the proposed time-varying framework successfully learns and adapts to these behaviors. The authors suggest future work should explore non-periodic non-stationary settings, different families of basis functions, and benchmarking against non-stationary transformer architectures to further clarify the trade-offs of this approach.