Technical Summary: End-to-End Identifiable and Consistent Recurrent Switching Dynamical Systems

Problem Statement

Learning identifiable representations in deep generative models for sequential data remains a fundamental challenge, particularly when the data exhibits regime-switching dynamics where underlying temporal mechanisms change over time. Existing approaches to Switching Dynamical Systems (SDSs) face two primary limitations:

1. Theoretical Limitations: Current identifiability results rely on restrictive assumptions, such as stationary latent dynamics or specific emission models (e.g., piecewise-linear). There is a known trade-off ("no free lunch") where relaxing constraints on emissions often necessitates stricter constraints on latent dynamics, and vice versa.
2. Practical Limitations: Most existing methods rely on Variational Autoencoder (VAE) estimators. These introduce approximation gaps due to the variational lower bound (ELBO) and amortized inference, which can hinder the accurate recovery of the true latent structure and limit disentanglement.

Furthermore, while non-recurrent switching models have been studied, recurrent switching dynamical systems (rSDSs)—where the switching process depends on past latent states—lack a comprehensive end-to-end identifiability framework that handles nonlinear transitions and invertible emissions simultaneously.

Methodology

The authors propose a theoretical framework and a practical estimator, $\Omega$SDS, to address these gaps.

1. Theoretical Framework: Identifiability of rSDS

The paper establishes identifiability for a broad class of recurrent switching dynamical systems under flexible assumptions. The model consists of:

• Emission Model (A1): Observations are generated via an invertible flow-based transformation of continuous latent variables and exogenous noise.
• Latent Dynamics (A2): Latent variables follow a regime-dependent Gaussian autoregressive process with analytic mean functions and diagonal covariances.
• Switching Process (A3): The discrete switching process is recurrent, meaning transition probabilities depend on both the previous switch and the previous latent state ($p(s_t | s_{t-1}, z_{t-1})$).

The authors prove three key identifiability results under specific regularity conditions (including "sticky" switching, transition separation, and covariance separation):

• Latent Identifiability: The latent representation is identifiable up to affine transformations.
• Transition Identifiability: The regime-dependent transition dynamics are identifiable up to permutations of the regimes.
• Disentanglement: By imposing regime-dependent variance assumptions (A8), the affine ambiguity is reduced to permutations and scaling, enabling fully disentangled representations.

2. Estimator: $\Omega$SDS

Motivated by the theory, the authors introduce $\Omega$SDS, a likelihood-based estimator designed to avoid the approximation gaps of VAEs.

• Architecture: It employs a flow-based emission (using TarFlow for synthetic data and Glow for video) to invertibly map observations to latent variables and noise.
• Inference: Instead of variational inference, it optimizes the exact marginal likelihood using Expectation-Maximization (EM). The E-step computes exact switch posteriors using forward-backward recursions on the latent rMSM, and the M-step updates parameters via gradient ascent.
• Initialization: To prevent state collapse (where only a subset of regimes is used), the method initializes rMSM parameters using PCA features and employs a PCA-alignment auxiliary loss early in training.

Key Contributions

1. End-to-End Identifiability Theory: The paper extends prior results by establishing identifiability for recurrent switching systems with nonlinear transition dynamics and invertible emissions. It proves that latent variables, transition dynamics, and the switching process can be recovered up to acceptable equivalences (affine transformations and permutations) without restricting emissions to be linear or stationary.
2. Consistent Estimation ($\Omega$SDS): The authors propose a flow-based estimator that optimizes the exact likelihood. Unlike VAE-based approaches, this method is consistent up to the identifiable equivalence class under standard Maximum Likelihood Estimation (MLE) conditions for recurrent Markov Switching Models.
3. Empirical Validation: The method is validated on synthetic data, 2D bouncing ball videos, and real-world dancing videos, demonstrating superior performance in disentanglement and forecasting compared to VAE-based baselines.

Experimental Results

The authors evaluated $\Omega$SDS against state-of-the-art baselines (including iSDS, NCTRL, CtrlNS, SNLDS, and VRNN) across three domains:

• Synthetic Data: In noiseless settings, $\Omega$SDS maintained consistently high Mean Correlation Coefficient (MCC) scores for latent recovery across increasing dimensions, whereas VAE-based methods degraded as dimensionality increased. In noise-augmented settings, $\Omega$SDS remained competitive, demonstrating scalability.
• 2D Bouncing Ball Videos: This benchmark tests recurrent switching where wall collisions (continuous states) influence regime switches.
  • Regime Estimation: The recurrent variant ($\Omega$SDS (R)) achieved perfect regime estimation (F1 = 1.000) on observed sequences and high accuracy (F1 = 0.932) on future predictions, outperforming non-recurrent models (iSDS, $\Omega$SDS (A)) which struggled with future dynamics.
  • Forecasting: $\Omega$SDS (R) achieved the lowest Long-Range Perceptual Image Patch Similarity (LPIPS) errors for long-horizon rollouts (500 steps), significantly outperforming VAE-based recurrent models like SNLDS and non-switching baselines.
• Real-World Dancing Videos (AIST): On complex human motion data, $\Omega$SDS produced more interpretable regime decompositions (e.g., distinguishing "move up" vs. "move down") and generated more plausible future dance movements compared to iSDS, which tended to produce stationary poses.

Significance and Claims

The paper claims that $\Omega$SDS provides a robust alternative to VAE-based SDS methods by bridging the gap between theoretical identifiability and practical estimation.

• Theoretical Impact: It resolves the "no free lunch" trade-off in previous SDS literature by showing that recurrent feedback in the switching process, combined with invertible emissions and specific variance structures, allows for full identifiability without restricting the emission map to be linear or the latent dynamics to be stationary.
• Practical Impact: By optimizing the exact likelihood rather than a variational bound, the method avoids the approximation errors that often plague disentanglement in deep generative models. The empirical results suggest that enforcing identifiable latent dynamics does not sacrifice predictive performance; rather, the recurrent structure enhances long-horizon forecasting and interpretability.
• Limitations: The authors modestly note that their approach requires constraining latent dynamics (specifically the likelihood-separation condition A5), which may not trivially generalize to arbitrary nonlinear dynamics. Additionally, the consistency result relies on standard MLE assumptions (e.g., global optimization, ergodicity) beyond the identifiability assumptions.

The work positions itself as a step toward more transparent modeling of complex time series, with potential applications in climate data, motion analysis, and behavioral modeling, where understanding the underlying switching mechanisms is critical.