How many unseen species are in multiple areas?

1. Problem Statement

The paper addresses the statistical challenge of estimating species richness and predicting the discovery of new species in heterogeneous populations across multiple areas (specifically two areas, $d=2$).

• Context: In ecology and other fields (e.g., genomics, text mining), researchers often need to estimate the total number of distinct species (richness) and the number of shared species (species present in both areas) based on observed samples.
• Limitations of Existing Methods:
  • Single-Area Focus: Most existing Bayesian nonparametric methods (e.g., based on Gibbs-type priors) are designed for a single homogeneous population.
  • Frequentist Constraints: Frequentist approaches for multiple areas often rely on asymptotic assumptions and are limited to one-step-ahead predictions (predicting the next single observation). They struggle to provide closed-form expressions for $m$-step-ahead predictions (predicting the number of new species in a future sample of arbitrary size $m$).
  • Computational Complexity: Previous Bayesian attempts for multiple areas often rely on computationally expensive Markov Chain Monte Carlo (MCMC) methods and lack closed-form estimators for shared species.

2. Methodology

The authors propose a novel framework based on the Vector of Finite Dirichlet Processes (Vec-FDP) prior, recently introduced by Colombi et al. (2024).

2.1 The Statistical Model

• Partial Exchangeability: The model assumes two sequences of observations, $X_1$ and $X_2$, from two different areas. The sequences are exchangeable within each group but not across groups.
• Vec-FDP Prior:
  • The underlying probability measures $(P_1, P_2)$ are modeled as a vector of dependent random probability measures.
  • Finite Support: Both areas share a common, finite, yet random number of species, denoted by $M$.
  • Shared Atoms: The species labels (atoms) $\tau_1, \dots, \tau_M$ are common to both areas.
  • Group-Specific Weights: The proportions (weights) of these species, $w_{j,m}$, vary between areas. Specifically, $w_j | M \sim \text{Dir}_M(\gamma_j, \dots, \gamma_j)$, where $\gamma_j$ controls the homogeneity within group $j$.
  • Prior on $M$: The total number of species $M$ follows a shifted Poisson distribution ($M \sim \text{Pois}_1(\Lambda)$).

2.2 Theoretical Derivations

The core contribution is the derivation of closed-form expressions for the partially Exchangeable Partition Probability Function (pEPPF) and subsequent predictive distributions.

• Coefficients $V$: The authors define coefficients $V^r_{n_1, n_2}$ that allow for the exact calculation of probabilities without integration.
• In-Sample Analysis: They derive the joint distribution of:
  • $K_{j, n_j}$: Local number of distinct species in area $j$.
  • $K_{n_1, n_2}$: Global number of distinct species (union of both areas).
  • $S_{n_1, n_2}$: Number of shared species (intersection).
• Out-of-Sample Prediction: They derive the joint posterior predictive distribution for the number of new species in a future sample of sizes $m_1$ and $m_2$. This includes:
  • New local species ($K^{(n_j)}_{j, m_j}$).
  • New global species ($K^{(n_1, n_2)}_{m_1, m_2}$).
  • New shared species ($S^{(n_1, n_2)}_{m_1, m_2}$).
• Key Insight: The framework allows for the prediction of species discovery for arbitrary future sample sizes ($m_1, m_2$), extending beyond the one-step-ahead limitation of frequentist estimators.

2.3 Parameter Estimation

Two strategies are proposed for estimating the hyperparameters $(\gamma_1, \gamma_2, \Lambda)$:

1. Bayes I (Maximum Marginal Likelihood): Maximizes the pEPPF.
2. Bayes II (Diversity-Based): Uses the relationship between the parameters and diversity indices (Simpson's index and Morisita index) to solve for parameters analytically. This method is computationally negligible compared to Bayes I.

3. Key Contributions

1. Closed-Form Predictive Theory: The paper provides the first closed-form Bayesian nonparametric solution for predicting unseen distinct and shared species in two areas. It avoids MCMC, enabling exact calculations.
2. $m$-Step-Ahead Prediction: Unlike frequentist methods restricted to one-step predictions, this framework predicts the number of new species for any future sample size $m$, facilitating sample size determination and stopping rules.
3. Joint Modeling of Shared Species: It explicitly models the dependence between areas, allowing for the estimation of shared species richness and the probability of discovering new shared species, a problem previously lacking tractable Bayesian solutions.
4. Computational Efficiency: The use of closed-form expressions allows predictions to be computed in less than a second, a significant improvement over MCMC-based alternatives.

4. Results

The methodology was validated through simulation studies and a real-world application.

4.1 Simulation Study

• Scenarios: Tested under various weight distributions (Dirichlet, Geometric, Zipf) and dependency structures (including cases where areas share no species).
• Performance:
  • Accuracy: The proposed estimators (Bayes I and Bayes II) achieved competitive or superior Root Mean Squared Error (RMSE) compared to "Independent" (ignoring dependence) and "Pooled" (ignoring group structure) benchmarks.
  • Robustness: The method remained robust even under model misspecification (e.g., Zipf weights vs. Dirichlet priors).
  • Shared Species: The method was the only one capable of accurately predicting the number of new shared species, a metric where competitors failed or were undefined.
  • Computational Speed: Bayes II was significantly faster for parameter estimation as sample sizes increased, as it avoids calculating the full marginal likelihood.

4.2 Real-World Application (Ants in Trieste)

• Dataset: Ant populations sampled from two distinct locations in Trieste, Italy: a semi-natural forest (Bosco Bovedo) and a city park (Orto Lapidario).
• Findings:
  • The model successfully estimated the number of shared and distinct species.
  • Stopping Rules: Using $m$-step-ahead heatmaps, the authors demonstrated that the probability of discovering new species drops rapidly as current sample size increases. This provides a practical tool for researchers to decide when to stop sampling.
  • Comparison: The model provided non-zero probabilities for discovering new shared species, whereas frequentist competitors (Yue & Clayton, Chao et al.) often yielded zero or highly uncertain intervals, suggesting the sample was exhausted when it might not be.

5. Significance and Future Directions

• Ecological Impact: The framework offers a rigorous, computationally efficient tool for biodiversity assessment, allowing researchers to optimize sampling efforts and accurately estimate the "unseen" portion of biodiversity in multiple habitats.
• Theoretical Advancement: It bridges the gap between single-area Bayesian nonparametrics and multi-area problems, introducing a tractable dependent prior structure.
• Future Work:
  • Scalability: Extending the theory to $d > 2$ areas is challenging due to the exponential growth of combinatorial statistics (pairwise, triple-wise overlaps).
  • Approximations: Developing asymptotic approximations for the $V$ coefficients to handle very large datasets.
  • Applications: Applying the framework to multi-armed bandit problems for optimal sampling allocation.

In summary, this paper presents a major advancement in Bayesian nonparametrics by solving the complex problem of shared species discovery in multiple areas with exact, closed-form solutions, offering both theoretical depth and practical utility for ecological and statistical research.