💻 computer science

The M-Tensor Format: Optimality in High Dimensional Regression for Nonlinear Models with Scarce Data

This paper introduces an M-Tensor regression framework that leverages tensor algebra and kernel properties to effectively model high-dimensional nonlinear systems with scarce data, thereby overcoming the curse of dimensionality and enabling robust parameter estimation without fixed-point strategies.

Original authors: Rémi Cloarec, Sebastian Rodriguez, Xavier Kestelyn, Francisco Chinesta

Published 2026-02-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rémi Cloarec, Sebastian Rodriguez, Xavier Kestelyn, Francisco Chinesta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict the weather, but instead of looking at just temperature and humidity, you have to account for 300 different variables simultaneously (like wind speed at every street corner, cloud density in every cubic meter, etc.).

In the world of math and engineering, this is called a "high-dimensional" problem. Usually, trying to solve this with standard methods is like trying to fill a swimming pool with a teaspoon: the amount of data you need grows so fast (exponentially) that it becomes impossible. This is known as the "Curse of Dimensionality." If you have scarce data (which is often the case in expensive engineering tests), standard models simply break or give nonsense answers.

This paper introduces a new tool called the M-Tensor Format to solve this problem. Here is how it works, using simple analogies:

1. The Problem: The "Wall of Complexity"

Imagine you are trying to build a recipe for a cake.

Standard Method: You try to write down every single possible combination of ingredients (flour, sugar, eggs, baking powder, etc.) in one giant list. If you add just one more ingredient, your list doubles in size. With 300 ingredients, your list would be longer than the number of atoms in the universe. You can't store it, and you can't calculate it.
The Data Problem: You only have a few recipes (data points) to learn from. Trying to guess the rules for 300 ingredients based on 50 recipes is usually a recipe for disaster (overfitting).

2. The Solution: The "Lego Block" Approach (M-Tensor)

The authors propose a smarter way to build the model. Instead of one giant, unmanageable list, they use separated variables.

Think of the M-Tensor as a set of Lego blocks.

Instead of building one massive, solid block representing the whole cake, you build the model by stacking small, simple 1D Lego blocks (representing one variable at a time) together.
The "M-Tensor" is a special way of snapping these blocks together. It allows the computer to see the whole picture without ever having to build the giant, heavy block in its memory.
The Magic Trick: The paper uses a mathematical operation called the "m-tensor product." Imagine this as a special glue that holds your Lego blocks together. It allows the computer to do complex calculations by only touching the small blocks, rather than the whole structure. This keeps the memory usage low and the speed high, even when you have hundreds of variables.

3. Handling "Scarce Data": The "Filter" (Regularization)

Since we don't have enough data to perfectly define 300 variables, the model might get confused and start "hallucinating" patterns that don't exist. To fix this, the paper introduces Regularization techniques.

Think of this as a noise-canceling filter or a sieve:

Tikhonov & Spectral Truncation: These are like turning down the volume on the "faint whispers" in the data. They tell the model, "Ignore the tiny, shaky details that might just be random noise; focus only on the strong, clear signals."
ALI Regularization (The "Smart Subset"): This is the most creative part. Imagine you have a choir of 1,000 singers (your data points), but they all sound very similar. The ALI method listens to the choir and says, "We don't need all 1,000 singers. We only need these 50 unique voices to represent the whole song."
- It finds the "Almost Linearly Dependent" (ALD) rows—basically, it finds the redundant data and throws it away.
- This makes the model much faster to run later because it only has to remember a tiny fraction of the original data, while still keeping the accuracy.

4. The Connection to "Kernels" (The "Magic Mirror")

The paper explains that this method is secretly a type of Kernel Regression (a popular machine learning technique).

Usually, Kernel methods use a "magic mirror" to look at data in a higher dimension without actually building it.
The M-Tensor format builds this mirror explicitly using the Lego blocks (tensor products). It gets all the benefits of the "magic mirror" (handling complex, non-linear relationships) without needing to pick a specific "type" of mirror. It builds the mirror out of the data itself.

5. Does it Work? (The Proof)

The authors tested this on two famous challenges:

The Rosenbrock Function: A standard math test for optimization. They scaled it up to 300 dimensions. The M-Tensor method handled it easily, showing that the time it takes to solve the problem grows in a straight line (linear) rather than exploding exponentially.
Dynamical Systems (Lorenz Attractor & Kuramoto Oscillators): These are chaotic systems (like weather or coupled pendulums) that are very hard to predict.
- They used the method to predict how these systems move over time.
- Even with very little data, the M-Tensor models could predict the future behavior of systems with 100 variables with about 2% error.
- The "ALI" version was the fastest, running significantly quicker than the others because it had "pruned" the unnecessary data.

Summary

The M-Tensor Format is a new mathematical toolkit that lets engineers and scientists solve problems with hundreds of variables using very little data.

It breaks the "Curse of Dimensionality" by breaking big problems into small, manageable Lego-like pieces.
It saves memory and time by never building the giant, impossible matrix.
It filters out noise using smart selection techniques (ALI) to work with scarce data.

In short, it turns an impossible math problem into a manageable one by changing how the computer looks at the data, rather than just trying to throw more computing power at it.

Technical Summary: The M-Tensor Format for High-Dimensional Regression

1. Problem Statement

The paper addresses the challenge of performing nonlinear regression in high-dimensional contexts where data is scarce. In engineering and design optimization, problems often depend on a large number of parameters. Traditional approaches suffer from the "curse of dimensionality," where the number of model coefficients and the required data volume grow exponentially with the number of parameters. This renders standard matrix-based regression intractable for systems with hundreds of parameters when data is limited and costly to obtain.

Existing solutions, such as sparse grids or variable separation techniques like Proper Generalized Decomposition (PGD) and Tensor-Train (TT) formats, offer partial relief. However, many of these methods rely on fixed-point strategies (e.g., Alternating Least Squares) that construct solutions sequentially, potentially introducing suboptimal corrections late in the process. Furthermore, some tensor methods require complex algebraic structures or sequential algorithms (like TT-SVD) that can become computational bottlenecks. The authors seek a framework that preserves the algebraic properties of matrix-based regression while avoiding the curse of dimensionality and eliminating the need for fixed-point iterative strategies.

2. Methodology: The M-Tensor Format

The authors propose a regression framework based on a specific tensor algebraic structure called the m-tensor format. This approach leverages the separation of variables to represent high-dimensional functions as products of one-dimensional basis functions.

Core Concepts

Partial Tensor Product (m-product): The method introduces a "row-wise tensor product" (denoted as the m-tensor product). Unlike standard tensor products that flatten structures, this operation preserves the multidimensional structure of the data. It generalizes the face-splitting product (transposed Khatri-Rao product) but avoids the index mapping burdens associated with it.
Tensor Representation: A high-dimensional scalar function $f(x)$ is approximated as an inner product between a tensor of basis functions $\phi(x)$ and a coefficient tensor $C$ . The basis functions are constructed as the m-tensor product of one-dimensional basis vectors $\psi$ .
Least Squares Formulation: The regression problem is formulated to find the coefficient tensor $C$ $C$ that minimizes the $\ell_2$ $ℓ_{2}$ error. By utilizing the properties of the m-product, the authors derive an explicit solution for the coefficients without ever constructing the full, exponentially large coefficient tensor.
- The solution relies on the matrix $P = \Phi \ltimes \Phi^\top$ , where $\Phi$ is the m-tensor regression operator.
- The solution is computed via $\hat{C} = \Phi^\top (\Phi \ltimes \Phi^\top)^{-1} y$ , but implemented efficiently using Cholesky factorization to ensure numerical stability.
Kernel Interpretation: The framework is shown to be equivalent to kernel regression. The m-product effectively computes the kernel matrix $K$ where entries are inner products of the mapped basis functions. This allows the authors to leverage kernel properties (like Mercer's theorem) without explicitly defining a kernel function, as the tensor structure implicitly generates all combinations of the chosen one-dimensional bases.

Regularization Techniques

To address ill-conditioning and overfitting in scarce data scenarios, the paper proposes three regularization strategies adapted to the m-tensor format:

Tikhonov Regularization: Adds an $\ell_2$ penalty to the coefficients, effectively leveling up small singular values of the operator.
Spectral Truncation: Uses the Singular Value Decomposition (SVD) of the operator $P$ to truncate small singular values, projecting the solution onto a lower-dimensional subspace.
ALI (Almost Linearly Independent) Regularization: This is a key contribution. It selects a subset of "almost linearly independent" rows from the data matrix based on a distance criterion (similar to the ALD criterion in kernel recursive least squares). This reduces the system size from $m$ (total samples) to $\tilde{m}$ (selected samples), significantly lowering computational costs while maintaining a bounded error relative to the optimal SVD solution.

3. Key Contributions

Algebraic Minimalism: The paper presents an algebraically minimal approach to high-dimensional regression that generalizes matrix-based least squares without requiring the complex algebraic machinery of general tensor networks (like TT or Tucker formats).
Avoidance of Fixed-Point Strategies: Unlike PGD or CP-ALS methods, the m-tensor format provides a direct, non-iterative solution for the regression coefficients, ensuring optimality in the $\ell_2$ sense from the available data.
Scalability: The method demonstrates linear scaling with respect to the number of parameters ( $n$ ) for inference and manageable scaling (cubic in the worst case) for construction, effectively breaking the curse of dimensionality for regression tasks.
ALI Regularization: The introduction of ALI regularization allows for massive reductions in system size and computational cost during the online inference stage, particularly when the number of retained samples $\tilde{m}$ is much smaller than the total samples $m$ .

4. Numerical Results

The authors validate the framework on three benchmarks:

Rosenbrock Function Approximation: The method was tested on dimensions ranging from 20 to 300 with varying sample sizes. Results showed that the construction time scales polynomially (theoretically cubic, practically lower) and inference time scales linearly with dimension. The ALI regularization model demonstrated the ability to fully linearize the inference cost.
3D Lorenz Attractor: Used to test robustness in dynamical system identification. The m-tensor models (Least Squares, Spectral Truncation, and ALI) successfully predicted the system's trajectory over 25,000 time steps (50 times the training period) from random initial conditions. The ALI model showed good generalizability, though the authors note that further robustness could be achieved by minimizing the $\ell_0$ norm.
Kuramoto Oscillators: The framework was applied to a system of coupled oscillators with dimensions up to $n=100$ . Despite being massively underdetermined (1000 training steps for 100 parameters), the models maintained approximately 2% relative error. The ALI model retained significantly fewer rows (e.g., ~370 rows for $n=100$ ) compared to the full dataset, confirming its efficiency.

5. Significance and Claims

The paper claims that the m-tensor format offers a robust, efficient, and easy-to-implement alternative for high-dimensional regression in data-scarce environments.

Optimality: It provides an optimal solution in the $\ell_2$ sense without the sequential construction errors inherent in fixed-point strategies.
Efficiency: By preserving the tensor structure, the method avoids the memory and computational explosion associated with flattening high-dimensional problems into matrices.
Flexibility: The framework allows for the use of arbitrary one-dimensional basis functions and naturally incorporates kernel properties, enabling the use of techniques like kPCA without being restricted to a predefined set of kernel functions.
Practicality: The authors emphasize that the approach is "algebraically minimalistic," making it conceptually simpler than abstract kernel feature spaces or general tensor networks, while still achieving the necessary computational gains.

The authors conclude that while the method is currently implemented in a sequential NumPy environment, its structure is inherently parallelizable, suggesting significant potential for further performance gains. They position the m-tensor format as a promising tool for model order reduction, operator inference, and solving parametric PDEs.