Technical Summary: Beyond Additive Decompositions: Interpretability through Separability

Problem Statement

Interpretable machine learning faces a fundamental tension between predictive accuracy and structural transparency. While "black-box" models (e.g., deep neural networks, gradient boosting ensembles) achieve state-of-the-art performance, they lack readable structures. Conversely, "glass-box" models like Generalized Additive Models (GAMs) offer transparency but often fail to capture complex interactions or rely on misspecified parametric assumptions.

Current interpretability methods, including post-hoc explanations (e.g., SHAP, Integrated Gradients) and glass-box models, largely rely on additive decompositions. The authors argue that additive representations suffer from critical failure modes in the presence of strong interactions:

1. Signal Cancellation: Local SHAP explanations can cancel out main and interaction effects, leading to zero attribution for active features.
2. Interaction Masking: Partial dependence plots (PDPs) marginalize over other features, effectively discarding higher-order interaction information and recovering only main effects.
3. Off-Support Extrapolation: Marginal averaging in correlated feature spaces can lead to spurious patterns in low-density regions.

The paper posits that these limitations stem from the inherent inability of additive projections to faithfully represent multiplicative or separable interaction structures without information loss.

Methodology: Tensor Separation Learning (TSL)

The authors propose Tensor Separation Learning (TSL), a regression model designed to learn a sum of rank-1 products of univariate per-feature functions. Unlike standard separable models, TSL enforces specific structural constraints to ensure stability and interpretability.

Model Formulation

The TSL estimator $\hat{m}(x)$ is defined as a sum of $R$ stages, where each stage $\ell$ is the difference of two positive separable products:
$$\hat{m}(x) = \sum_{\ell=1}^{R} \left( \lambda^{(\ell)}_+ \prod_{j=1}^p \hat{m}^{(\ell)}_{+,j}(x_j) - \lambda^{(\ell)}_- \prod_{j=1}^p \hat{m}^{(\ell)}_{-,j}(x_j) \right)$$
Key constraints include:

• Positivity: All component functions $\hat{m}^{(\ell)}_{\pm,j}(x_j)$ are constrained to be strictly positive ($>0$). This ensures that each component unambiguously amplifies or suppresses its product, removing sign ambiguity inherent in unconstrained products.
• Ordered Difference: Each stage is a difference of two positive terms, allowing the model to represent signed contributions while maintaining positive factors.

Parameterization: Backbone and Tilt

To enhance interpretability, the authors introduce a backbone/tilt reparameterization for each feature pair in a stage:
$$\hat{m}^{(\ell)}_{\pm,j}(x_j) = b^{(\ell)}_j(x_j) e^{\pm d^{(\ell)}_j(x_j)}$$

• Backbone ($b$): Represents the shared magnitude or "activity gate" of the stage. It determines where the stage is active (near-zero values switch the stage off).
• Tilt ($d$): Represents the signed imbalance between the positive and negative branches. It controls the directionality of the contribution.
  This decomposition separates the magnitude of an effect from its signed direction, allowing for faithful visualization even when the net signed contribution is near zero due to cancellation.

Fitting Procedure

TSL employs a stagewise greedy procedure with orthogonal refitting:

1. Stagewise Fitting: At each stage $\ell$, the model fits the current residuals using a grid-tensor refinement approach inspired by CART (Classification and Regression Trees). It learns piecewise-constant univariate functions on an adaptive partition.
2. Bagging and Aggregation: To reduce variance, multiple grid tensors are fitted in parallel on bootstrap samples. A similarity-based aggregation scheme (anchoring to a reference grid and filtering outliers) combines these into a single stage predictor.
3. Orthogonal Refitting: After fitting a new stage, all stage coefficients $\{\lambda^{(k)}_\pm\}$ are jointly optimized via least squares against the target $y$. This backfitting ensures the coefficients are optimal given the current set of stages.

Key Contributions

1. Tensor Separation Learning (TSL): A novel regression algorithm that estimates the conditional mean as a sum of differences of positive rank-1 separable products.
2. Model-Native Interpretability: The authors prove that for TSL, the fitted model can be exactly reconstructed from its first-order partial dependence functions (up to constant factors). Unlike post-hoc methods, TSL's visualizations (1D partial dependence curves) are not approximations but faithful representations of the fitted components. The backbone/tilt decomposition allows practitioners to distinguish between magnitude (activity) and direction (signed imbalance).
3. Approximation Theory: The paper establishes an $O(1/\sqrt{r})$ approximation rate for the TSL hypothesis class on functions with bounded mixed $p$-th order partial derivatives (specifically in the Sobolev space $W^{(1,1)}_{mix}$), utilizing the Orthogonal Greedy Approximation (OGA) framework.

Experimental Results

The authors evaluated TSL on the OpenML CTR 23 regression suite (27 datasets) against state-of-the-art baselines:

• Baselines: Black-box models (XGBoost, LightGBM, Random Forest) and interpretable models (Explainable Boosting Machines (EBM), SepALS—a separable model without TSL's constraints).
• Performance: TSL achieves competitive predictive performance. In the interpretable group, TSL ranks in the top 3 on 17 of 27 datasets and is the best interpretable model on 5.
• Comparative Analysis:
  • TSL outperforms EBM and tree ensembles on datasets with strong low-rank separable structures (e.g., socmob, California housing).
  • On California housing, TSL captures localized, non-monotone spatial interactions (e.g., coastal premiums) that EBM and XGBoost smooth out or fail to isolate in 1D summaries.
  • Synthetic Case Study: In a synthetic example where a strong quadratic effect is masked by a third-order interaction (resulting in a zero 1D partial dependence), TSL successfully recovers the underlying magnitude via its backbone curves, whereas standard PDPs and ICE curves fail to reveal the signal without complex aggregation.
  • Ablation: Removing the positivity constraint and ordered-difference structure (TSL 1-product) significantly degrades performance, confirming that these constraints are crucial for representational flexibility and stability.

Significance and Claims

The paper claims that TSL offers a distinct path to interpretability by moving "beyond additive decompositions." Its primary significance lies in:

• Faithful Visualization: By enforcing separability and positivity, TSL ensures that partial dependence plots recover the exact shape of the fitted factors, avoiding the information loss and masking inherent in additive projections of interactive models.
• Handling Interactions: TSL provides a mechanism to visualize and understand high-order interactions through 1D summaries (backbone and tilt) without resorting to opaque black-box explanations or high-dimensional plots.
• Competitive Accuracy: It demonstrates that a glass-box model with a separable structure can compete with powerful tree-ensemble black boxes on regression benchmarks, challenging the notion that high accuracy requires sacrificing interpretability.

The authors remain modest regarding limitations, acknowledging that TSL's approximation rates depend on mixed smoothness assumptions and that the separable representation suffers from non-identifiability (requiring the specific aggregation/filtering procedures described). They note that on purely additive problems, existing methods like EBM may remain stronger baselines.