On a joint simultaneous learning of relevant feature subsets and subspaces in regression-like problems

Technical Summary: Entropy-Optimal Manifold Regression (EOMR)

Problem Statement
The paper addresses the challenge of modeling and predicting non-stationary and nonlinear regression problems, specifically within the context of chaotic dynamical systems and fluid dynamics. A primary difficulty identified is the "small data learning challenge," where the feature space dimension ($D$) is large relative to the available observation statistics ($T$). This scenario is common in fields like fluid mechanics (due to expensive numerical simulations) and climate research (due to short observation histories).

The author argues that state-of-the-art Artificial Intelligence (AI) and Machine Learning (ML) tools, including gradient-boosted random forests, deep neural networks (NN), and transformer-based models (e.g., TabPFN), struggle in these regimes. These methods often suffer from:

1. Overfitting: They tend to memorize training data rather than learn generalizable patterns, resulting in poor performance on test data (often worse than trivial "persistent" predictors).
2. Computational Scalability: Transformer-based architectures scale quadratically with both feature dimension and data size, making them computationally prohibitive for large, realistic problems.
3. Model Complexity: These tools often require a number of tunable parameters comparable to or exceeding the size of the training dataset, leading to high memory and iteration costs.

Methodology: Entropy-Optimal Manifold Regression (EOMR)
The paper proposes Entropy-Optimal Manifold Regression (EOMR), a supervised extension of the previously introduced unsupervised Entropy-Optimal Manifold Clustering (EOMC). EOMR aims to jointly identify relevant feature subsets and subspaces for regression tasks.

• Mathematical Formulation:
  The model assumes $K$ "local" mappings from a feature vector $X(:, t)$ to a scalar target $Y(t)$. Each mapping is characterized by:

  • An orthogonal $d$-dimensional linear manifold projector $T_k$ (subspace reduction).
  • A diagonal weight matrix $W$ (subset selection), where diagonal elements $w_i$ represent the probability that feature $i$ is relevant.
  • Regression coefficients $\tilde{\theta}_k$.

  The prediction is formulated as a weighted expectation over these local linear models, regularized by entropy terms. The learning objective is to minimize a constrained optimization problem ($L_{EOMR}$) that includes:

  • A least-squares error term for regression.
  • A metrization term (variance-like regularization) to ensure manifold projection stability.
  • Entropic regularization terms ($\phi_1$ for model probabilities $\gamma$, and $\phi_2$ for feature weights $W$) to enforce sparsity and unbiasedness (maximum entropy principle).

• Numerical Solution:
  The optimization is solved via a subspace-iteration algorithm. The problem is decomposed into five steps, alternating between updating variables while freezing others:

  1. Manifold Projectors ($T_k$): Solved via Riemannian gradient descent on the Stiefel manifold.
  2. Centroid Vectors ($\mu_k$): Solved analytically as sample-weighted means.
  3. Regression Coefficients ($\tilde{\theta}_k$): Solved analytically via regularized normal equations.
  4. Model Probabilities ($\gamma$): Solved analytically using a softmax function derived from the Euler-Lagrange principle.
  5. Feature Weights ($W$): Solved using a Sequential Projected Gradient algorithm based on a Fast Entropy Approximation (FEA).

• Complexity:
  The algorithm exhibits linear scaling in both iteration time and memory complexity with respect to the feature dimension $D$ and data size $T$ (specifically $O(KDT)$), provided the subspace dimension $d \ll D$. This contrasts sharply with the quadratic scaling of transformer-based methods.

Key Contributions

1. Joint Learning: EOMR simultaneously learns relevant feature subsets (via $W$) and subspaces (via $T_k$), unifying dimensionality reduction and feature selection in a single probabilistic framework.
2. Scalability: The method offers linear computational and memory scaling, making it suitable for high-dimensional, small-data problems where deep learning and transformers are infeasible or inefficient.
3. Interpretability and Sparsity: By utilizing entropic regularization, EOMR produces sparse, entropy-optimal models. It identifies the minimal set of features and parameters required to describe the system dynamics.
4. Robustness in Chaos: The method is designed to handle non-stationary and nonlinear dynamics without assuming global linearity or stationarity, yet it can distill simple, causal, and weakly-stationary autoregressive structures when they exist.

Experimental Results
The paper evaluates EOMR on two challenging benchmarks:

1. Lorenz-96 System (F=8 and F=12):

   • Task: Predicting the leading Essential Orthogonal Function (EOF) of the system in strongly and very-strongly chaotic regimes.
   • Comparison: EOMR was compared against Lasso-regularized multilinear regression, Random Forests/XGBoost, Deep Neural Networks, and TabPFN v3.0.
   • Findings: Deep learning and tree-based methods performed worse than a trivial "persistent" predictor on test data, indicating severe overfitting. EOMR achieved a Root Mean Squared Error (RMSE) of $\approx 7 \times 10^{-6}$, which was orders of magnitude lower than competitors. Furthermore, EOMR models were significantly sparser (approx. 100 parameters) compared to the complexity of the training data, whereas other models required parameters exceeding the dataset size.

2. Modified Hasegawa-Wakatani (mHW) Model:

   • Task: Predicting drift-wave turbulence dynamics in the edge of a tokamak plasma.
   • Findings: Similar to the Lorenz-96 results, standard AI tools (NN, RF/XGB, TabPFN) failed on test data, performing worse than the persistent predictor. EOMR achieved an RMSE of $\approx 4.5 \times 10^{-4}$, nearly an order of magnitude better than the persistent predictor and significantly better than Lasso MLR.
   • Model Distillation: EOMR distilled the complex dynamics into a simple linear AutoRegressive Moving Average (ARMA) model with only 8 tunable parameters. The identified model was proven to be weakly-stationary and causal (roots of the characteristic polynomial outside the unit circle).
   • Efficiency: A single EOMR prediction took $\approx 2$ nanoseconds on a standard laptop, which is 160 times faster than the physical lag of the process being predicted. In contrast, TabPFN took $\approx 0.02$ seconds (five orders of magnitude slower than the physical lag).

Significance and Claims
The paper claims that EOMR represents a "game-changer" for modeling chaotic and fluid dynamic systems under small-data constraints. Its significance lies in:

• Paradigm Shift: It challenges the "double-descent" trend of increasing model capacity and complexity, advocating instead for entropy-optimal, sparse, and simple models that generalize better.
• Practical Utility: The extreme computational efficiency (nanosecond-scale predictions) enables real-time online prediction, optimization, and control of fast chaotic systems, which is currently impossible with heavy transformer-based models.
• Robustness: It demonstrates that for specific classes of complex systems, the "best" model is not the most complex one, but the one that correctly identifies the underlying low-dimensional, sparse structure, avoiding the memorization pitfalls of data-hungry AI.

The author concludes that EOMR provides a robust, interpretable, and computationally feasible alternative to current state-of-the-art AI tools for regression in non-stationary, nonlinear, and chaotic environments.