Technical Summary: Perturbation-based Effect Measures for Compositional Data

Problem Statement

Compositional data, where observations consist of parts of a whole summing to a constant (typically normalized to the simplex $\Delta_{d-1}$), are ubiquitous in fields such as microbiome research, ecology, and social sciences. Existing methods for assessing the effect of compositional features on a response variable $Y$ face significant limitations:

1. Inadequacy of Traditional Parametric Approaches: Many modern applications involve high-dimensional and sparse data (e.g., microbiome species counts), which are poorly modeled by traditional parametric log-ratio approaches.
2. Bias in Marginal Analyses: Assessing how summary statistics of a composition (e.g., a diversity index) affect a response is often misleading. Because the components of a composition are deterministically tied by the sum-to-one constraint, marginal analyses fail to account for confounding induced by the remaining variation in the composition.
3. Interpretability vs. Flexibility Trade-off: Flexible nonparametric machine learning methods can model the relationship between $Y$ and $Z$ but often lack interpretable effect measures. Conversely, interpretable measures (like log-contrast coefficients) often rely on strict parametric assumptions that may not hold.
4. The Zero Problem: Standard log-ratio transformations require data to be in the open simplex (no zeros), necessitating ad-hoc zero-imputation strategies that can distort results.

The authors argue that defining interventions on compositional variables is difficult because individual components are not free-standing; changing one component necessitates changes in others to satisfy the simplex constraint.

Methodology

The paper proposes a unified framework based on hypothetical data perturbations to define interpretable statistical functionals called Average Perturbation Effects.

1. Perturbation Framework

Instead of transforming the data or assuming a specific regression model, the authors define effects directly on the simplex via perturbations $\psi(z, \gamma)$, which map a composition $z$ to a trajectory in the simplex.

• Binary Perturbations: Encode discrete changes (e.g., setting a component $Z_j$ to 0 and rescaling the rest). The effect is the average contrast in the response between the original and perturbed states, normalized by the probability of change.
• Directional Perturbations: Encode continuous changes (e.g., moving towards the center of the simplex to increase diversity). The effect is the average initial rate of change of the regression function along the perturbation path.

2. Derivative-Isolating Reparametrization

A core technical contribution is the construction of a derivative-isolating reparametrization $\phi: \Delta_{d-1} \to \mathbb{R} \times \mathcal{W}$.

• This bijection transforms the compositional vector $Z$ into a scalar variable $L$ and a nuisance vector $W$.
• Crucially, the perturbation $\psi$ only changes $L$ while leaving $W$ invariant.
• This transformation reduces the problem of estimating a complex functional on the simplex to estimating a standard average partial effect (for directional perturbations) or a conditional mean contrast (for binary perturbations) in a generic setting, allowing the application of established semiparametric theory.

3. Estimation Strategies

The paper proposes efficient estimators for these effects using semiparametric techniques:

• Nonparametric Estimation: Uses one-step corrected estimators (based on influence functions) with cross-fitting to estimate the target parameters. This requires estimating nuisance functions (regression functions, propensity scores, or conditional score functions).
• Partially Linear Estimation: Assumes a partially linear model $E[Y|L, W] = \theta L + h(W)$. Here, $\theta$ corresponds exactly to the perturbation effect. This approach relies only on estimating conditional means ($E[Y|W]$ and $E[L|W]$), which is generally more robust and less sensitive to estimation errors than estimating derivatives or score functions.
• Handling Zeros: The framework naturally handles zeros in the data. For perturbations where the speed vanishes at the boundary (e.g., multiplicative speed when $Z_j=0$), the estimators are adapted to exclude these points from the derivative calculation without requiring imputation.

4. Specific Perturbation Constructions

The authors provide explicit constructions for practical applications:

• Compositional Feature Influence (CFI): Measures the effect of increasing a specific component $Z_j$. They propose a "multiplicative speed" (scale-invariant) and a "unit speed."
• Compositional Knock-out Effect (CKE): A binary perturbation measuring the effect of removing a component (setting it to 0).
• Compositional Diversity Influence (CDI): Measures the effect of increasing diversity (moving towards the simplex center). They propose a "Gini-speed" based on the Gini coefficient to ensure the effect is interpretable in terms of diversity indices.

Key Results

The paper validates the proposed framework through theoretical analysis and empirical experiments:

1. Theoretical Guarantees: The authors prove uniform asymptotic normality for their estimators under standard semiparametric conditions. They show that the reparametrization allows for valid inference without needing to model the complex geometry of the simplex directly.
2. Simulation Studies:
   • In semi-synthetic US Census data, marginal analyses of racial diversity on income produced significant negative effects due to confounding by the remaining compositional structure. The proposed CDI correctly identified the positive effect constructed in the data.
   • In microbiome simulations, standard feature importance measures (permutation importance, nonparametric $R^2$) failed because they violate the simplex constraint by holding other coordinates fixed. The perturbation-based measures remained consistent across different regression algorithms.
3. Real-World Applications:
   • New York Schools: Analyzing the effect of racial diversity on student grades. The marginal approach suggested a strong positive effect even after controlling for covariates. The CDI approach, which accounts for the compositional structure, found smaller positive effects or effects indistinguishable from zero, suggesting the marginal results were driven by compositional confounding.
   • American Gut Microbiome: Predicting BMI from species abundance. The proposed CFI (multiplicative speed) and CKE identified different sets of important microbes compared to standard log-contrast regression. The log-contrast coefficients were shown to be highly sensitive to the choice of pseudocounts, whereas the perturbation-based measures (specifically CKE) explicitly handle zeros and distinguish between presence/absence effects and continuous abundance effects.

Significance and Claims

The authors claim that their work provides a unified framework for defining interpretable effect measures for compositional features that:

• Avoids the Zero Problem: Unlike log-ratio approaches, it does not require data transformation or zero imputation.
• Handles High Dimensions: It is compatible with flexible, nonparametric machine learning methods via semiparametric estimation, making it suitable for sparse, high-dimensional data like microbiome datasets.
• Controls for Confounding: By defining effects through perturbations that respect the simplex constraint, it naturally controls for the confounding induced by the remaining variation in the composition, which marginal analyses of summary statistics fail to do.
• Bridges Causality and Statistics: While the effects are defined as statistical quantities, the framework provides a clear path to causal interpretation by formalizing component-level questions (e.g., "increase diversity") as well-defined interventions on the entire composition.

The paper positions these "Average Perturbation Effects" as a necessary alternative to existing methods that either rely on restrictive parametric assumptions or produce misleading results by ignoring the compositional nature of the data.