Robust Tensor-on-Tensor Regression

1. Problem Statement

Tensor-on-Tensor (TOT) regression is a statistical framework used to predict a response tensor ($Y$) from a set of predictor tensors ($X$). While standard TOT regression (e.g., using Least Squares with CP decomposition) is effective for capturing multi-way dependencies, it suffers from two critical vulnerabilities in real-world applications:

1. Sensitivity to Outliers: Standard methods are highly sensitive to both casewise outliers (entire observations that deviate from the bulk) and cellwise outliers (individual anomalous entries within a tensor). Cellwise outliers are particularly prevalent in high-dimensional tensor data (e.g., images, where a few pixels may be corrupted).
2. Missing Values: Real-world tensor data often contain missing entries. Standard approaches either discard incomplete observations (losing data) or use simple imputation that does not account for the tensor structure or outlier contamination.

Existing robust methods typically address only one type of outlier (either casewise or cellwise) or only one side of the regression (predictors or response). There was no unified method capable of simultaneously handling casewise and cellwise outliers in both predictors and responses, while also accommodating missing values.

2. Methodology: The ROTOT Framework

The authors propose ROTOT (Robust Tensor-on-Tensor regression), a novel method that integrates robust statistics with multilinear algebra. The methodology consists of three main components:

A. Handling Predictors (ROMPCA)

To address outliers and missing values in the predictor tensors $\{X_n\}$, the method employs Robust Multilinear Principal Component Analysis (ROMPCA).

• Mechanism: ROMPCA decomposes the predictor tensors into a core tensor and projection matrices using a robust loss function.
• Output: It produces:
  • Imputed Tensors ($X^{imp}_n$): Replacing cellwise outliers and missing values with robust estimates.
  • Casewise Weights ($w^x_n$): A weight of 0 for casewise outliers and 1 for regular observations.
  • Core Tensors: Used to calculate score distances for leverage point detection.

B. Robust Loss Function for the Response

For the response tensors $\{Y_n\}$, ROTOT minimizes a weighted M-estimation objective function that simultaneously targets cellwise and casewise deviations. The objective function $L$ is defined as:
$$L = \frac{\hat{\sigma}_2^2}{2m} \sum_{n=1}^N m_n w^x_n \rho_2 \left( \frac{1}{\hat{\sigma}_2} \sqrt{\frac{1}{m_n} \sum_{q} m_{n,q} \hat{\sigma}_{1,q}^2 \rho_1 \left( \frac{r_{n,q}}{\hat{\sigma}_{1,q}} \right)} \right) + \lambda \|B\|_F^2$$
Where:

• $r_{n,q}$ represents the cellwise residuals.
• $\rho_1$ (Cellwise): A bounded function (specifically the hyperbolic tangent, $\tanh$) applied to individual cell residuals. This limits the influence of cellwise outliers.
• $\rho_2$ (Casewise): A bounded function applied to the aggregated deviation of an entire observation. This limits the influence of casewise outliers.
• Scale Estimators: Robust M-scale estimators ($\hat{\sigma}_1$ and $\hat{\sigma}_2$) are used to standardize residuals, ensuring the method remains robust even if the majority of data is contaminated.
• Missing Values: Handled via a mask tensor $M_n$ that excludes missing entries from the summation, effectively treating them similarly to cellwise outliers but with known positions.

C. Optimization Algorithm (IRLS)

Since the objective function is non-convex, the authors propose an Iteratively Reweighted Least Squares (IRLS) algorithm:

1. Initialization: Uses a two-step strategy involving the Detecting Deviating Cells (DDC) algorithm and a preliminary robust fit to avoid local minima.
2. Alternating Least Squares (ALS): In each iteration, the algorithm updates the slope tensor parameters (mode matrices $U_\ell$ and $V_m$) and the intercept $B_0$ by solving weighted least squares problems derived from the first-order conditions.
3. Weight Updates: Weights are recalculated based on the current residuals and scale estimates.
4. Convergence: The algorithm is proven to monotonically decrease the objective function until convergence.

3. Key Contributions

1. First Unified Approach: ROTOT is the first TOT regression method capable of simultaneously handling casewise outliers, cellwise outliers, and missing values in both predictor and response tensors.
2. Dual Robustness: By nesting a cellwise robust loss ($\rho_1$) within a casewise robust loss ($\rho_2$), the method prevents a single cellwise outlier from triggering a false casewise outlier flag, a common issue in previous robust regression methods.
3. Diagnostic Tools: The authors introduce graphical diagnostics:
   • Residual Cellmap: A heatmap visualizing cellwise outliers in the response.
   • Outlier Map: A scatter plot of Residual Distance vs. Score Distance (from ROMPCA), colored by the percentage of outlying cells. This distinguishes between vertical outliers (bad response), leverage points (bad predictors), and good leverage points.
4. Software Implementation: The method is implemented in R, with code made publicly available.

4. Results

The performance of ROTOT was evaluated through extensive simulations and a real-world application:

• Simulation Study:

  • Scenarios: Tested under varying Signal-to-Noise Ratios (SNR) and contamination levels (5% to 10% of data) involving cellwise outliers, casewise outliers, and missing values.
  • Metrics: Relative Prediction Error (RPE) on clean validation sets.
  • Findings: ROTOT consistently outperformed classical TOT, "OnlyCell-TOT" (robust to cellwise only), and "OnlyCase-TOT" (robust to casewise only).
    • In mixed contamination scenarios, ROTOT achieved the lowest RPE.
    • Classical TOT and OnlyCase-TOT failed significantly when cellwise outliers were present in the response.
    • ROTOT maintained stability even with 5% missing values.

• Real Data Application (Labeled Faces in the Wild - LFW):

  • Task: Predicting 72 facial attributes (e.g., "Smiling", "Blurry", "Glasses") from face images (30x30 grayscale tensors).
  • Performance: ROTOT achieved a lower trimmed Mean Squared Error (robMSE) compared to classical TOT.
  • Robustness Check: When artificial outliers were injected into the attributes, ROTOT's performance remained stable, while classical TOT's accuracy degraded as contamination magnitude increased.
  • Diagnostics: The outlier map successfully identified specific individuals with corrupted attributes (e.g., "Blurry" and "Flash" for one subject) and individuals with obscured faces (casewise outliers).

5. Significance

The ROTOT method addresses a critical gap in multivariate data analysis. As data becomes increasingly high-dimensional and structured (tensors), the assumption of clean, complete data is rarely met.

• Practical Impact: It enables reliable modeling in fields like medical imaging (fMRI/EEG), computer vision, and chemometrics, where data corruption and missingness are common.
• Theoretical Advancement: It demonstrates how to extend robust M-estimation to the complex, non-linear optimization landscape of tensor decompositions without sacrificing computational feasibility.
• Interpretability: The proposed diagnostic tools allow practitioners not just to get robust predictions, but to understand the nature of the data anomalies (distinguishing between bad data points and bad model fits).

In summary, ROTOT provides a comprehensive, robust, and diagnostic-rich framework for tensor regression, ensuring reliability in the presence of the complex noise structures typical of modern tensor data.