📊 statistics

Distributional Extrapolation for Interactions

This paper introduces DExtrI, a method with theoretical guarantees for predicting combinatorial interaction effects in scenarios where training data contains only single active covariates, enabling successful generalization to unseen multi-covariate combinations in applications like drug discovery and hyperparameter optimization.

Original authors: Marin Šola, Xinwei Shen, Peter Bühlmann

Published 2026-08-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Marin Šola, Xinwei Shen, Peter Bühlmann

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In many fields of science and engineering, researchers face a stubborn limitation: they can easily measure how a system reacts to a single change, but they cannot afford to test every possible combination of changes. Imagine a pharmaceutical company trying to find the most effective drug cocktail. Testing every possible mix of dozens of compounds on human cells would take years and cost billions. Instead, scientists often test each drug alone, one by one, to see how it affects the cells. This creates a dataset that is rich in information about individual ingredients but completely silent on how those ingredients interact when mixed together. The central challenge is to predict the behavior of the whole mixture based solely on the behavior of its isolated parts. This is not just a matter of guessing; it is a mathematical puzzle where the answer lies in the hidden structure of how variables combine. If the relationship between the parts and the whole follows specific rules, it might be possible to reconstruct the unseen combinations from the known singles.

A team of researchers at ETH Zürich and the University of Washington has developed a new method called DExtrI to solve this exact problem. Their work focuses on a scenario where training data consists only of "axis-aligned" samples—measurements where only one variable is active at a time, while the test data involves multiple variables acting simultaneously. In the real world, this mirrors the drug discovery process where single-drug responses are abundant, but combination data is scarce. The researchers found that by modeling the outcome not as a simple average of parts, but as a complex interaction where a hidden noise factor enters the equation before the final calculation, they could mathematically prove that the entire system is recoverable. They demonstrated that if the underlying rules of the system are stable and follow certain structural patterns, the conditional distribution of the outcome for any new combination can be uniquely determined from the single-variable data. This means the model can generate plausible predictions for drug combinations it has never seen, effectively filling in the gaps of the experimental map.

The method works by treating the outcome as a sum of individual effects plus a set of interaction terms. Crucially, these interaction terms are not just added at the end; the noise or randomness of the system is injected into the argument of the interaction function before it is calculated. This subtle shift allows the model to "see" the interaction effects even when the data only shows one variable changing at a time. The researchers proved that under specific conditions—such as the interaction functions being non-polynomial and the noise affecting the system in a strictly monotonic way—the direction and shape of these interactions can be identified with certainty. They showed that once these components are learned from the single-variable data, the model can extrapolate to any combination of those variables, even those far outside the range of the original training data.

To test their theory, the team applied DExtrI to real-world datasets, including large-scale screens of drug combinations and hyperparameter optimization tasks for machine learning models. In the drug discovery experiments, they used data from thousands of single-drug tests to predict the viability of cells under complex drug cocktails. The results showed that DExtrI consistently outperformed standard machine learning models, which typically fail when asked to predict outcomes for combinations they have never encountered. For instance, on a dataset involving 38 different drugs and 39 cancer cell lines, the new method achieved significantly lower prediction errors than traditional approaches. Similarly, in the realm of artificial intelligence, where tuning multiple settings simultaneously is costly, the method successfully predicted the performance of complex configurations based on data where only one setting was changed at a time.

The study also explored settings where the data was not perfectly aligned with the axes but was close to it, such as predicting bike-sharing demand based on weather and time variables. Even in these "approximately axis-aligned" scenarios, the method held its ground, often providing more accurate forecasts than established baselines. The researchers noted that the approach is particularly robust in low-data regimes, where traditional models tend to overfit or fail to generalize. By relying on a distributional loss function that matches the entire shape of the predicted outcome rather than just the average value, the method captures the full uncertainty of the system. This allows it to generate not just a single point estimate, but a full range of possible outcomes, which is vital for risk assessment in fields like medicine.

While the method shows great promise, the authors are careful to note its boundaries. The mathematical guarantees rely on the assumption that the underlying system follows specific structural rules, such as the interaction functions being non-polynomial and the noise being unbounded. If these conditions are violated, the unique identification of the interactions may not hold. However, in the synthetic experiments designed to test these limits, the method consistently recovered the true interaction patterns where other models failed. The work suggests that one-factor-at-a-time experiments, often dismissed as insufficient for understanding complex systems, actually contain far more information about interactions than previously assumed. By unlocking this hidden information, DExtrI offers a powerful tool for scientists who need to navigate the vast space of possibilities without the luxury of testing every single option.

Technical Summary: Distributional Extrapolation for Interactions (DExtrI)

1. Problem Definition

The paper addresses the challenge of combinatorial extrapolation: predicting outcomes for combinations of covariates that were never jointly observed during training. Specifically, the authors focus on a setting where training data is axis-aligned, meaning each sample contains only one active covariate (e.g., single-drug responses in pharmacology or single-gene perturbations in genomics), while test-time inputs involve multiple simultaneously active covariates (e.g., drug combinations or joint hyperparameter configurations).

Standard machine learning models, including neural networks and empirical risk minimization (ERM), often fail in this regime because they struggle to generalize outside the convex hull of the training distribution. The authors note that acquiring fully combinatorial data is often prohibitively expensive, necessitating methods that can infer interaction effects from marginal, axis-aligned observations.

2. Methodology: The DExtrI Framework

The authors propose DExtrI (Distributional Extrapolation for Interactions), a generative regression framework that models the response variable $Y$ as a sum of additive components and interaction terms with a specific noise structure.

2.1 Model Structure

The model is defined as:
$Y = \sum_{j=1}^p f_j(X_j) + \sum_{k=1}^K g_k(\beta_k^\top X + h_k(\varepsilon))$
where:

$f_j$ are univariate additive functions for individual covariates.
$g_k$ are non-linear ridge functions representing interactions.
$\beta_k$ are coefficient vectors defining the interaction directions.
$h_k(\varepsilon)$ is a pre-additive noise term, where $\varepsilon \sim \mathcal{N}(0, 1)$ is a shared latent noise variable.

2.2 Key Mechanism: Pre-Additive Noise

The core innovation is the use of pre-additive noise (noise added inside the argument of the interaction function $g_k$ ) rather than the standard post-additive noise.

Generative Interpretation: The model specifies the full conditional distribution $Y|X=x$ as a deterministic transformation of a low-dimensional latent noise variable $\varepsilon$ .
Extrapolation Logic: Because $h_k(\varepsilon)$ has full support on $\mathbb{R}$ , the argument $\beta_k^\top X + h_k(\varepsilon)$ spans $\mathbb{R}$ even when $X$ is restricted to axis-aligned supports. This allows the model to "see" the behavior of the interaction function $g_k$ across its entire domain through the variation in $\varepsilon$ , even if the training data only covers specific marginal slices of $X$ .

2.3 Estimation

DExtrI estimates the model components by minimizing the Energy Score, a strictly proper scoring rule for probability distributions.

Objective: Minimize the expected energy score between the observed conditional distribution and the model's generated distribution.
Implementation: The components $f_j, g_k, h_k$ are parameterized using neural networks. The optimization uses the Adam algorithm.
Advantage: Unlike ERM which minimizes point-wise squared error, the energy score matches distributions via pairwise distances, which the authors argue is more robust in low-sample-size settings and crucial for identifying the structural components from axis-aligned data.

3. Theoretical Contributions

The paper establishes rigorous theoretical guarantees regarding the identifiability and extrapolability of the model.

3.1 Identifiability

The authors prove that under specific structural assumptions (e.g., $g_k$ are non-polynomial exponential polynomials, $h_k$ are strictly monotone with distinct asymptotic growth orders, and interaction supports are distinct), the model is identifiable from axis-aligned data.

Result: If two parameterizations induce the same conditional distribution on the axis-aligned training support, they must be equivalent up to trivial indeterminacies (scaling of $\beta_k$ and $h_k$ , and additive constants).
Significance: This is a non-trivial result because the training support has an empty interior (it lies only on the axes), preventing the use of standard mixed derivatives to identify interaction directions. The pre-additive noise and monotonicity assumptions are shown to be necessary for disentangling the ridge directions and profiles.

3.2 Extrapolability

Based on the identifiability results, the paper proves that the learned model can deterministically extrapolate to the Cartesian product of the marginal supports ( $X_{test}$ ).

Theorem: If the model is identifiable, the learned function $\tilde{T}$ matches the true function $T$ on the entire test support $X_{test}$ (and beyond), not just in distribution but pointwise for fixed noise realizations.
Implication: The method can recover the full response surface for unseen combinations of covariates, provided the test points lie within the product of the observed marginal ranges.

4. Empirical Results

The authors evaluate DExtrI on synthetic and real-world datasets, comparing it against Neural Additive Models (NAM), standard ERM (MLP), and Engression (a related distributional regression method).

4.1 Synthetic Benchmarks

In high-dimensional synthetic settings ( $p=40$ ) with known ground truth, DExtrI consistently achieves significantly lower test Mean Squared Error (MSE) than baselines, often by orders of magnitude. Baselines fail to recover interaction effects from marginal data, while DExtrI successfully generalizes to the interior of the input space.

4.2 Real-World Applications

Drug Combination Screens (O'Neil and ALMANAC datasets): The method predicts the viability of cell lines under drug combinations using only single-drug training data. DExtrI outperforms baselines on the O'Neil dataset and performs competitively on ALMANAC, demonstrating utility in predicting untested drug combinations.
Hyperparameter Optimization (HPOBench): The method predicts validation error rates for MLP configurations based on marginal sweeps of individual hyperparameters. DExtrI achieves the lowest test MSE on 7 out of 8 tasks, showing superior ability to model joint hyperparameter interactions.
Approximately Axis-Aligned Data: The method is also tested on datasets where covariates are not strictly axis-aligned but concentrated near axes (e.g., Seoul Bike Sharing, Gas Turbine). DExtrI maintains robust performance, often outperforming ERM and NAM.

5. Significance and Claims

The paper claims that DExtrI enables combinatorial extrapolation as an identifiable statistical problem rather than solely a prediction challenge.

Information Content: A key implication is that "one-factor-at-a-time" (axis-aligned) experiments contain substantially more information about interactions than commonly assumed, provided the correct structural assumptions and distributional modeling are applied.
Practical Impact: The method offers a pathway to reduce the cost of exhaustive combinatorial experimentation in domains like drug discovery and hyperparameter tuning.
Limitations: The authors acknowledge that the method relies on specific structural assumptions (e.g., non-polynomial profiles, distinct growth orders of warps). They note that if these assumptions are violated (e.g., polynomial profiles without additional constraints), identifiability may fail. The paper does not claim to solve extrapolation for arbitrary distributions or supports outside the Cartesian product of marginals.

In summary, DExtrI provides a theoretically grounded framework that leverages distributional regression and pre-additive noise to recover interaction effects from limited, axis-aligned data, successfully generalizing to unseen covariate combinations in both synthetic and real-world scenarios.