Bayesian nonparametric models for zero-inflated count-compositional data using ensembles of regression trees

1. Problem Statement

The paper addresses the statistical challenges inherent in analyzing count-compositional data, which arises in fields such as ecology, palaeoclimatology, and microbiome research (e.g., pollen counts, microbial abundances). These data are characterized by multivariate counts constrained to sum to a fixed total. The authors identify four critical, often simultaneous, challenges that existing methods fail to address comprehensively:

1. Zero-Inflation: An excess of zeros beyond what is expected from standard sampling variability. Crucially, these zeros can be structural (the event cannot occur) or sampling (the event occurs but is not observed). Existing models often conflate these or fail to model structural zeros explicitly in a multivariate context.
2. Overdispersion: Variance significantly higher than that predicted by the multinomial distribution.
3. Complex Covariate Effects: Relationships between covariates (e.g., climate, diet) and compositions are often nonlinear and involve high-order interactions, which parametric linear models fail to capture.
4. Cross-Sample Heterogeneity: Unobserved latent factors causing dependence among categories and samples.

Existing approaches typically rely on parametric assumptions (e.g., linear functional forms) or specific distributions (e.g., Dirichlet-Multinomial) that restrict flexibility, or they use Gaussian Processes (GPs) which are computationally expensive and scale poorly with sample size.

2. Methodology

The authors propose two novel Bayesian nonparametric models based on ensembles of regression trees (BART): ZANIM-BART and ZANIM-LN-BART.

Core Framework: ZANIM Distribution

The models extend the Zero-and-N-inflated Multinomial (ZANIM) distribution (Menezes et al., 2025). This distribution treats the data as a finite mixture:

• Structural Zeros: Modeled via latent indicators $z_{ij}$ (Bernoulli) representing whether category $j$ in sample $i$ is structurally absent.
• Sampling Zeros: Arise naturally when the event occurs but count is zero.
• N-Inflation: Handles cases where zeros co-occur in all but one category.

The Proposed Models

1. ZANIM-BART:

   • Compositional Probabilities ($\theta$): Modeled using a Multinomial Logistic BART prior (Murray, 2021). This allows the population-level count probabilities to depend on covariates via a flexible, nonparametric sum of regression trees.
   • Structural Zero Probabilities ($\zeta$): Modeled using a Probit BART prior (Chipman et al., 2010). This allows the probability of a structural zero to depend on covariates nonparametrically.
   • Result: Captures nonlinearities and interactions for both the abundance and the "absence" mechanism without pre-specifying functional forms.

2. ZANIM-LN-BART:

   • Extends ZANIM-BART by incorporating multivariate logistic-normal random effects ($u_i$) into the compositional probabilities.
   • Purpose: To capture overdispersion and complex dependence structures among categories that cannot be explained by covariates alone.
   • Identifiability: Uses a sum-to-zero constraint and a factor-analytic hyperprior (Bhattacharya & Dunson, 2011) to parsimoniously model the covariance matrix of the random effects, avoiding the need for a fixed number of latent factors.

Inference Algorithm

The authors develop an efficient Markov Chain Monte Carlo (MCMC) algorithm combining:

• Data Augmentation: Adapts schemes from Menezes et al. (2025) and Albert & Chib (1993) to handle the zero-inflation and probit links. This introduces latent variables ($\phi_i$, $z_{ij}$, $w_{ij}$) to factorize the likelihood across categories.
• Bayesian Backfitting: Utilizes established BART sampling routines (Metropolis-Hastings for tree topologies, Gibbs sampling for terminal node parameters) to update the tree ensembles.
• Elliptical Slice Sampling: Used for updating the random effects in the ZANIM-LN-BART model.

3. Key Contributions

• Novel Nonparametric Framework: First application of BART to count-compositional data, explicitly modeling both structural zeros and compositional probabilities with flexible, data-driven functional forms.
• Unified Handling of Challenges: Simultaneously addresses zero-inflation (structural vs. sampling), overdispersion (via random effects), and complex covariate effects (via BART).
• New Model Derivations:
  • Introduces ZANIM-LN-BART as a generalization of ZANIM.
  • Derives MLN-BART (Multinomial Logistic-Normal BART) as a special case (when structural zeros are omitted), which appears to be a novel contribution to the literature.
• Computational Efficiency: Demonstrates that BART-based inference is computationally feasible and often more scalable than Gaussian Process alternatives for large sample sizes, while maintaining high flexibility.
• Robustness: Shows that the models perform well even when the data generating process is misspecified (e.g., when structural zeros are absent, the model does not degrade significantly).

4. Results

Simulation Studies

• Scenario 1 (Nonlinearity): In settings with nonlinear covariate effects and structural zeros, ZANIM-BART and ZANIM-LN-BART significantly outperformed competitors (ML-BART, MLN-GP, ZIDM-reg, DM-reg) in recovering true probabilities (measured by Kullback-Leibler divergence). Competing models suffered from bias when ignoring structural zeros or assuming linearity.
• Scenario 2 (Scalability & Complexity): Tested with varying dimensions ($d=20, 40$) and sample sizes ($n=200, 1000$). The proposed models consistently achieved lower KL divergence and better coverage probabilities (CP) for structural zeros and individual-level probabilities.
• Computational Cost: While BART models are slower than simple regression models, they are competitive with or faster than GP-based models (MLN-GP) as sample size increases, due to the poor scalability of GPs.

Real Data Applications

1. Palaeoclimate Reconstruction (Pollen Data):
   • Analyzed 7,832 samples of 28 pollen taxa against climate covariates.
   • ZANIM-LN-BART provided the best fit, accurately capturing high overdispersion, zero-inflation (63% zeros), and complex pollen-climate relationships (including multimodality).
   • Partial dependence plots revealed nuanced, nonlinear climate effects on specific taxa (e.g., Picea and Pinus.D) that parametric models missed.
2. Human Gut Microbiome:
   • Analyzed 96 samples with 13 taxa and 108 dietary covariates.
   • The model successfully performed variable selection and identified nonlinear dietary effects (e.g., the effect of Glycemic Index on Prevotella).
   • It outperformed ZIDM and DM regression models in posterior predictive checks, particularly in reproducing zero-inflation patterns.

5. Significance

This paper represents a significant advancement in the statistical analysis of compositional count data. By integrating BART with zero-inflated mixture models and latent random effects, the authors provide a "one-size-fits-all" framework that:

• Removes the need for restrictive parametric assumptions about covariate effects.
• Explicitly distinguishes between "true absence" (structural zeros) and "non-detection" (sampling zeros), which is critical for ecological and biological inference.
• Offers a computationally viable alternative to Gaussian Processes for large-scale, high-dimensional compositional data.

The work bridges the gap between flexible machine learning techniques (BART) and rigorous probabilistic modeling for compositional data, offering a powerful tool for researchers in palaeoclimatology, microbiome science, and ecology.