State-Space Neural Network with Ordered Variance for Model Order Determination

Technical Summary: State-Space Neural Network with Ordered Variance for Model Order Determination

Problem Statement
The paper addresses the challenge of identifying nonlinear state-space models from input-output training datasets while simultaneously determining an adequate model order (state dimension). While State-Space Neural Networks (SSNNs) have gained popularity for learning nonlinear dynamics, they suffer from issues common to Prediction Error Methods (PEMs), including convergence to local minima and, critically, the unknown state dimension. Selecting the model order typically requires trial-and-error, which is computationally expensive. Existing methods for model order determination in autoencoders or kernel-based models have not been effectively extended to SSNNs.

Methodology: SSNNO Framework
The authors propose a novel framework termed State-Space Neural Network with Ordered Variance (SSNNO). The core innovation is the integration of a variance-regularization term into the SSNN loss function to enforce an ordering of state variances.

1. Architecture and Training:

   • The SSNNO retains the standard SSNN structure, learning state ($x_{k+1} = f_{NN}(x_k, u_k)$) and output ($y_k = g_{NN}(x_k)$) functions via neural networks.
   • The loss function ($J$) combines the Squared Prediction Error (SPE), a parameter regularization term for the output subnetwork, and a novel variance regularization term ($J_v$).
   • The variance term penalizes the sample variances of the predicted states, weighted by a diagonal matrix $W = \text{diag}(w_1, \dots, w_d)$ where weights are strictly increasing ($0 \le w_1 < w_2 < \dots < w_d$).
   • This weighting scheme encourages the optimization process to minimize the variances of the trailing states (those with higher weights) to near-zero levels while preserving the variances of the leading states.

2. Model Order Determination and Reduction:

   • Significant vs. Residual States: After training, states are ordered by variance ($V_{x_1} \ge V_{x_2} \ge \dots \ge V_{x_d}$). A tolerance threshold $\delta$ is applied to classify states as "significant" ($V_{x_i} > \delta$) or "residual" ($V_{x_i} \le \delta$).
   • Reduced-Order SSNNO (R-SSNNO): The model order is reduced by retaining only the significant states. The residual states are approximated by their mean values and absorbed into the model parameters of the first and last layers of the state subnetwork. This yields a reduced-order model without requiring retraining.

3. Practical Implementation:

   • Recognizing that local optimization algorithms may not converge to the global minimum required for strict variance ordering, the authors propose a practical iterative algorithm (Algorithm 1). This method permutes the state variables of a locally optimized solution to satisfy the variance ordering condition, re-optimizing if necessary, until a variance-ordered solution is achieved.

Theoretical Contributions
The paper provides formal theoretical guarantees regarding the proposed framework:

• Existence of Ordered Variance: Using the rearrangement inequality, the authors prove that the global minimizer of the proposed loss function inherently satisfies the variance-ordering condition ($V_{x_1} \ge \dots \ge V_{x_d}$).
• Error Bound: The authors establish an upper bound on the output prediction error of the SSNNO relative to a standard SSNN. They demonstrate that the degradation in prediction accuracy due to variance regularization can be bounded by a user-specified tolerance, controlled by the hyperparameters $\alpha$ and $\beta$.

Simulation Results
The effectiveness of the SSNNO was validated on a nonlinear Continuous Stirred-Tank Reactor (CSTR) process, which exhibits severe gain nonlinearity.

• Model Order Identification: In simulations with a true system order of 2, the SSNNO (trained with an initial dimension of 3) successfully identified the third state as having negligible variance. This allowed for the derivation of a reduced-order (2nd order) R-SSNNO model.
• Performance: The R-SSNNO models (reduced to 2nd and 1st order) maintained prediction accuracy comparable to the full 3rd-order SSNNO and the standard SSNN, as evidenced by Mean Squared Error (MSE) metrics on training and testing data.
• Robustness: Monte Carlo simulations across varying noise levels confirmed that the variance ordering is consistently achieved without compromising the Squared Prediction Error (SPE).
• Control Application: The identified R-SSNNO model was successfully integrated into an Extended Kalman Filter (EKF)-based Model Predictive Control (MPC) scheme. The controller effectively tracked reference temperature trajectories, correcting for plant-model mismatch.

Significance and Claims
The paper claims that SSNNO is the first work to incorporate variance-ordering of state variables directly within an SSNN framework to solve the model order determination problem. Its primary significance lies in:

1. Simultaneous Identification: It enables the simultaneous identification of the nonlinear state-space model and the determination of an adequate model order from data alone.
2. Systematic Reduction: It provides a systematic, non-heuristic procedure to derive reduced-order models that retain essential dynamics while minimizing complexity.
3. Theoretical Rigor: It offers formal proofs for the existence of such ordered models and bounds on prediction error, addressing the lack of theoretical guarantees in many data-driven identification methods.

The authors conclude that while the method is effective for identification and control, future work is needed to extend the framework to provide theoretical guarantees on controllability and stability.