Intrinsic Dimension Estimating Autoencoder (IDEA) Using CancelOut Layer and a Projected Loss

Technical Summary: Intrinsic Dimension Estimating Autoencoder (IDEA)

Problem Statement
The paper addresses the challenge of estimating the intrinsic dimension (ID) of datasets where samples lie on linear or nonlinear manifolds within a high-dimensional embedded space. While numerous methods exist to estimate ID, many suffer from sensitivity to hyperparameters, high sample requirements, poor performance in high dimensions, or an inability to handle nonlinear geometries. Furthermore, classical estimators often provide only a dimension estimate without the ability to reconstruct the original data or offer an interpretable latent space. The authors specifically target the application of these methods to vertically resolved shallow flow simulations, where reducing high-dimensional numerical solutions to their intrinsic dimension is critical for efficiency, yet linear reduction techniques like Proper Orthogonal Decomposition (POD) may lose accuracy.

Methodology
The authors propose the Intrinsic Dimension Estimating Autoencoder (IDEA), a neural network architecture designed to simultaneously estimate the ID and reconstruct the data.

• Architecture: IDEA utilizes a standard autoencoder structure with an encoder, a bottleneck, and a decoder. The encoder and decoder consist of five fully connected linear layers with normalization and SiLU activation functions.
• Re-weighted Double CancelOut Layers: The core innovation lies in the bottleneck, which employs two successive "CancelOut" layers adapted from Borisov et al. (2019).
  • The first layer (Co1) acts as a sparsity-inducing gate. It applies a one-to-one mapping where each latent dimension is multiplied by a trainable weight. Through $L_1$ regularization, the model is encouraged to drive these weights to zero, effectively suppressing redundant latent variables.
  • The second layer (Co2) is unregularized and serves to rescale the remaining informative features, compensating for magnitude suppression caused by the regularization of Co1.
• Projected Loss Function: The training objective is defined by a composite loss function (Eq. 3) comprising four terms:
  1. Reconstruction Loss ($L_{rec}$): Standard Mean Squared Error (MSE) between the input and the full reconstruction.
  2. Projected Reconstruction Loss: A penalty term calculated by setting the last non-zero weight of Co1 to zero (simulating the removal of a dimension) and measuring the reconstruction error of this "projected" output. This term guides the model to determine if a dimension is truly necessary; if the loss does not significantly increase upon removal, the dimension is deemed redundant.
  3. $L_1$ Regularization: Encourages sparsity in the Co1 weights to minimize the effective number of active dimensions ($l_{eff}$).
  4. Orthogonality Penalty: A term based on the Frobenius norm of the difference between the correlation matrix of the latent space and the identity matrix, promoting disentanglement of latent features.
• Training Strategy: The model is trained to minimize the total loss, effectively finding the smallest $l_{eff}$ that preserves reconstruction quality. The estimated ID ($\tilde{d}$) corresponds to the number of non-zero weights in Co1 after training.

Key Contributions

1. IDEA Framework: The introduction of an autoencoder that integrates ID estimation directly into the reconstruction process via controlled sparsity in the bottleneck.
2. Projected Loss Term: A novel loss component that evaluates reconstruction quality under the hypothetical removal of a latent dimension, providing a dynamic mechanism to identify the minimal sufficient dimensionality.
3. Re-weighted Double CancelOut: A specific architectural adaptation that allows for the suppression of dimensions while maintaining the scale of remaining features, facilitating stable convergence.
4. Application to Fluid Dynamics: The successful application of IDEA to vertically resolved free-surface flow simulations, demonstrating its ability to capture nonlinear dependencies that linear methods (POD) miss.

Results

• Synthetic Benchmarks: IDEA was tested on synthetic datasets generated from Legendre polynomial combinations and standard benchmark manifolds (e.g., Swiss Roll, helices) from Campadelli et al. (2015).
  • IDEA accurately predicted the intrinsic dimension for linear and simple nonlinear manifolds.
  • On complex nonlinear manifolds (e.g., the 1D helix embedded in 3D), IDEA sometimes estimated a slightly higher dimension (e.g., 2 instead of 1) due to topological constraints, though it still outperformed or matched many state-of-the-art estimators (MLE, TwoNN, lPCA) which showed significant errors on specific datasets.
  • The model demonstrated high reconstruction accuracy, with test losses remaining low when using the estimated $\tilde{d}$ dimensions.
• Free-Surface Flow Application:
  • Case 1 (Simple Flow): For a flow with negligible friction, IDEA identified an intrinsic dimension of 3 (including water height), implying a 2D manifold for velocity profiles. It achieved comparable accuracy to a 4-moment Legendre decomposition using only 2 latent variables for the velocity field.
  • Case 2 (Complex Flow): For a flow with significant friction and complex vertical dynamics, IDEA estimated an ID of 3 (2 for velocity). It successfully reconstructed the solution using only 2 latent variables, whereas POD required 5 modes and a moment model required 7 coefficients to achieve similar accuracy.
  • Interpretability: The latent variables showed clear correlations with physical quantities (water height and Legendre coefficients), suggesting the model learned physically meaningful representations, unlike standard autoencoders or POD which can be less interpretable.

Significance and Claims
The paper claims that IDEA offers a versatile and robust approach to intrinsic dimension estimation that overcomes several limitations of classical estimators. By directly providing an integer-valued ID and a corresponding reconstruction capability, IDEA enables efficient dimensionality reduction without sacrificing accuracy. The authors highlight that unlike linear reduction techniques (POD) or standard moment models, IDEA can exploit nonlinear correlations between variables to represent complex physical systems with fewer parameters. The method is presented as a data-driven tool that requires no explicit knowledge of the underlying generative process, making it suitable for analyzing complex numerical simulations where the governing equations are known but the solution structure is high-dimensional and nonlinear. The authors note that while the model is robust, performance on highly complex nonlinear manifolds may benefit from advanced learning rate scheduling, as demonstrated in their ablation studies.