Technical Summary: Sparse Deep Additive Model with Interactions (SDAMI)

Problem Statement

The paper addresses the challenge of regression in small-$n$, large-$k$ regimes (few observations, many predictors), a setting common in scientific domains like neuroscience where interpretability is as critical as predictive accuracy. While deep learning excels in data-rich settings, standard deep neural networks (DNNs) often fail in small-sample scenarios due to overfitting and lack of interpretability. Conversely, traditional sparse additive models (SAMs) offer interpretability but primarily capture univariate structures, often missing interaction-driven signals, particularly when interactions exist without strong associated main effects (pure interactions).

Existing interpretable methods often rely on heredity constraints (requiring main effects to be present before an interaction can be selected), which limits their ability to detect pure or higher-order interactions. The authors identify a gap in methods that can flexibly model complex interactions while maintaining effect-level interpretability and avoiding the computational infeasibility of exhaustive interaction searches (which scale as $\binom{k}{2}$).

Methodology: SDAMI Framework

The authors propose the Sparse Deep Additive Model with Interactions (SDAMI), a three-stage framework designed to discover and estimate main and interaction effects without imposing heredity constraints. The core innovation is the Effect Footprint principle, which posits that higher-order interactions leave detectable marginal traces on constituent variables even in the absence of strong main effects.

The Three-Stage Strategy

1. Stage 1: Effect Footprint Screening

   • Goal: Identify a candidate set of variables ($\hat{S}$) that includes both true main effects and variables involved in interactions (footprint variables).
   • Mechanism: The method fits an augmented sparse additive model using SpAM (Sparse Additive Models) with B-spline bases. It screens for variables where the estimated function $\hat{f}_j \neq 0$.
   • Theoretical Basis: Based on the Effect Footprint definition ($m_j(x) = E[f(X_I) | X_j = x]$), the authors prove that a variable leaves a non-zero marginal footprint unless it satisfies rare, degenerate symmetry conditions (e.g., perfect antisymmetry in independent, mean-centered inputs). This allows the reduction of the search space from $\binom{k}{2}$ pairs to $\binom{s}{2}$, where $s \ll k$.

2. Stage 2: Group Lasso Decomposition

   • Goal: Disentangle main effects from interactions within the screened set $\hat{S}$.
   • Mechanism: The method constructs grouped basis expansions for the screened variables. It forms univariate blocks for main effects and pairwise tensor-product blocks for interactions. A Group Lasso penalty is applied to select groups of coefficients.
   • Outcome: This yields two sets: $\hat{M}$ (selected main effects) and $\hat{I}$ (selected interaction pairs). Theoretical analysis confirms this stage achieves effect-level selection consistency (oracle recovery) without heredity constraints.

3. Stage 3: Constrained Deep Regression

   • Goal: Flexible, nonlinear estimation of the selected effects.
   • Mechanism: Dedicated deep subnetworks are assigned to the selected components:
     • A univariate subnetwork $NN^{(j)}$ for each $j \in \hat{M}$.
     • A shared multivariate subnetwork $NN^{(I)}$ for the interaction set $\hat{I}$.
   • Constraints: Sparsity and interpretability are enforced via norm-based constraints on the first-layer weights ($\|W^{(1)}\|_\infty \leq \kappa \|f\|$). If an effect is deemed zero (based on Stage 2), the corresponding weights are pruned. This ensures the network structure aligns with the recovered effect structure.

Key Contributions

1. The SDAMI Framework: A principled, three-stage approach for sparse additive-plus-interaction modeling in high-dimensional settings that does not require heredity constraints.
2. The Effect Footprint Principle: A theoretical derivation showing that interaction-bearing variables leave detectable marginal traces (footprints) under general conditions, enabling the discovery of pure interactions. The authors prove footprints vanish only under measure-zero symmetry conditions.
3. Theoretical Guarantees:
   • Theorem 4.1: Characterizes the detectability of footprints, proving they are robust to noise and correlation in practice.
   • Theorem 4.2: Establishes effect-level selection consistency for the screening and decomposition stages, proving the method recovers the true active set with probability tending to one.
   • Theorem 4.3 & 4.4: Prove prediction convergence in probability and provide finite-sample prediction bounds for the final estimator.
4. Empirical Performance: Demonstrated superior performance over state-of-the-art interpretable models (e.g., NAM, GAMI-Net, NODE-GA2M) and standard DNNs in simulations and real-world benchmarks.

Experimental Results

The authors evaluated SDAMI on six synthetic scenarios (varying main effect strength, overlap, and pure interaction presence) and six real-world datasets (including V1 fMRI, Diabetes, and Chip lifetime data).

• Predictive Accuracy: SDAMI consistently achieved the lowest or near-lowest Root Mean Squared Error (RMSE) across all settings. In Case 6 (Pure Interactions), where heredity-constrained models (NAM, GAMI-Net) failed significantly (RMSE ~0.97 vs. SDAMI's 0.46), SDAMI successfully recovered the structure.
• Variable Selection: SDAMI maintained a high True Positive Rate (TPR) while achieving a near-zero False Positive Rate (FPR). In contrast, baselines like LASSONET and SODA showed steep drops in sensitivity for interaction-only cases or introduced spurious discoveries.
• Real-World Application:
  • V1 fMRI: SDAMI identified interpretable interaction surfaces between Gabor features, revealing non-additive complex-cell responses that align with biological hypotheses.
  • Chip Lifetime & Diabetes: The model successfully identified biologically and physically meaningful interactions (e.g., between BMI and serum triglycerides) while filtering out noise features.
• Robustness: Additional experiments in the supplementary material showed SDAMI remains competitive under correlated covariates, heteroscedastic noise, and binary response settings.

Significance and Claims

The paper positions SDAMI as a structured screening-and-estimation framework tailored specifically for high-dimensional interaction discovery where exhaustive search is infeasible.

• Interpretability vs. Flexibility: The authors claim SDAMI successfully bridges the gap between the flexibility of deep learning and the interpretability of additive models. By separating main and interaction effects into dedicated subnetworks, it provides effect-level visualizations (component functions) that support scientific hypothesis generation.
• Overcoming Heredity: A primary significance is the ability to detect pure interactions (where no main effect exists) without relying on the heredity assumption, a limitation of many existing structured sparse methods.
• Scientific Utility: The framework is presented as a tool for scientific discovery in fields like neuroscience, where understanding how variables interact is as important as prediction accuracy. The authors emphasize that SDAMI is not a universal replacement for all interpretable models but a specialized solution for the small-$n$, large-$k$ regime with complex interaction structures.

The authors acknowledge limitations, noting that the footprint principle may weaken under exact symmetry (e.g., binary predictors) and that future work could explore higher-order interactions and computational scalability via SIS-based screening.