Handling Missingness and Censoring in Dirichlet Models

Technical Summary: Handling Missingness and Censoring in Dirichlet Models

Problem Statement
Compositional data, which represent relative proportions of parts of a whole and reside on a unit simplex, are frequently encountered in fields ranging from microbiology to environmental science. The Dirichlet distribution is the canonical probabilistic model for such data, analogous to the multivariate normal distribution for unconstrained data. However, a significant limitation in current practice is that likelihood-based inference for Dirichlet parameters typically requires fully observed compositions. In real-world scenarios, data often contain missing components or censored values (e.g., due to detection limits or incomplete speciation).

Existing approaches to handle this incompleteness generally fall into three categories: deletion of incomplete rows (which discards information), substitution with fixed values (e.g., zeros), or imputation. Imputation methods often rely on transforming data to an unconstrained space (e.g., via log-ratio transformations), fitting a model, and mapping results back to the simplex. This transformation-based approach suffers from several drawbacks: it can render original variable interpretations unrecoverable, fail to provide injective mappings back to the simplex, and struggle to naturally accommodate censoring mechanisms. Furthermore, non-parametric imputation methods often ignore the specific censoring structure, leading to potential bias and distorted underlying distributions.

Methodology
The authors propose a unified framework that treats both missing and censored components as forms of "coarsening" within the sense of Heitjan & Rubin (1991). This framework encompasses Missing Completely at Random (MCAR), Missing at Random (MAR), and Missing Not at Random (MNAR) as special cases of Coarsening Completely at Random (CCAR), Coarsening at Random (CAR), and Coarsening Not at Random (CNAR), respectively. The paper focuses on the CAR mechanism, where the coarsening depends on observed data but not unobserved exact values, rendering the mechanism ignorable for likelihood-based inference.

The core of the methodology is an Expectation–Maximisation (EM) type algorithm designed to perform Maximum Likelihood (ML) estimation of Dirichlet parameters directly on the simplex without data transformation. The algorithm proceeds as follows:

1. Theoretical Foundation: The authors derive the marginal density of observed components and the conditional distribution of unobserved components given the observed ones. They establish that the conditional distribution of unobserved components, given the observed parts and the feasible region defined by censoring bounds, follows a truncated Dirichlet distribution.
2. E-Step: The algorithm computes the conditional expectation of the complete-data log-likelihood. This involves calculating the expected value of the logarithm of the unobserved components. Using the properties of the truncated Dirichlet distribution, the authors derive a closed-form expression for these expectations involving the digamma function and the derivative of the normalizing constant of the truncated distribution.
3. M-Step: The algorithm maximizes the expected complete-data log-likelihood with respect to the parameter vector $\alpha$. To ensure the positivity of the Dirichlet parameters during optimization, the authors employ a reparameterization ($\alpha_k = e^{\beta_k}$) and utilize a Newton–Raphson (NR) algorithm nested within the EM iterations. The Hessian matrix is inverted efficiently using the Sherman–Morrison formula.
4. Convergence: The algorithm employs specific stopping rules: an inner rule based on the Newton–Raphson convergence and an outer rule based on the Aitken acceleration criterion applied to the observed-data log-likelihood.

Key Contributions
The paper makes several specific technical contributions:

• Unified Framework: It develops a single likelihood-based framework that simultaneously handles missing and censored components on the simplex, avoiding the need for data transformation.
• EM Algorithm for Dirichlet: It derives a computationally efficient EM-type algorithm for ML estimation of Dirichlet parameters in the presence of incomplete data, including the derivation of necessary conditional expectations involving truncated Dirichlet distributions.
• Minimal Data Requirements: The method allows for the fitting of a Dirichlet distribution even when a dataset has rows with only a single observed component, a scenario where many existing transformation-based methods fail.
• Joint Estimation and Imputation: The approach performs parameter estimation and model-based imputation jointly, preserving the compositional structure and avoiding the errors associated with back-transformation.

Results and Evaluation
The performance of the proposed method was evaluated through two primary avenues:

1. Simulation Study: An extensive simulation study (detailed in the Supplementary Material) assessed the method under increasingly complex coarsening mechanisms. The results indicated that the proposed estimator substantially improves parameter recovery compared to ML estimation based only on complete cases. Furthermore, the imputations generated by the proposed algorithm exhibited smaller errors than those produced by an existing model-based approach (ALR-EM) and a non-parametric alternative (geometric mean imputation). The method demonstrated reliability even under extreme conditions, such as datasets with up to 90% unobserved values.

2. Real Data Application: The methodology was applied to mercury speciation data from the National Health and Nutrition Examination Survey (NHANES). This dataset contained 3,872 observations with approximately 72% of cells unobserved due to detection limits and incomplete speciation.

   • The Dirichlet model provided a better fit to the observed data than the multivariate logistic-normal model (used in the ALR-EM approach), as evidenced by higher observed log-likelihood, AIC, and BIC values.
   • Wasserstein distance calculations confirmed that the fitted Dirichlet and logistic-normal distributions model the data differently, with the Dirichlet providing a more plausible representation.
   • The method successfully estimated the proportions of unknown mercury types and provided imputations that better preserved the observed compositional structure compared to competing approaches.

Significance and Claims
The authors claim that this work addresses a fundamental gap in compositional data analysis: the lack of a likelihood-based estimation method that directly accommodates incomplete proportions on the unit simplex. By reversing the traditional perspective of adapting data to unconstrained models, the proposed method develops estimation tools that respect the geometry and constraints of compositional data.

The significance of the work lies in its ability to enable likelihood-based inference and clustering directly on the simplex in the presence of unobserved values. The authors suggest that this opens the door to further developments in simplex-based modelling, including richer mixture structures and more flexible inferential procedures. The method is presented as a robust alternative that avoids the interpretational limitations and potential biases introduced by transformation-based approaches, particularly in scenarios involving high levels of missingness or censoring.