📊 statistics

Handling Missingness and Censoring in Dirichlet Mixture Models

This paper proposes a novel Expectation-Maximisation algorithm for fitting finite mixtures of Dirichlet distributions directly on the simplex to handle missing and censored compositional data, demonstrating through simulations and real-world applications that this approach outperforms traditional case-deletion methods in clustering accuracy and model selection while preserving data interpretability.

Original authors: Jason Pillay, Andriette Bekker, Cristina Tortora, Antonio Punzo

Published 2026-07-31

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Jason Pillay, Andriette Bekker, Cristina Tortora, Antonio Punzo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to describe a smoothie, but you can only taste a few of the fruits inside. Maybe you know there's strawberry and banana, but the blender hid the mango and the kiwi. In the world of data science, this is called "compositional data." It's information that describes how parts make up a whole, like the ingredients in a recipe or the chemicals in a rock. The tricky part is that these parts are locked together: if you have more of one ingredient, you automatically have less of the others, because they must always add up to 100%. Scientists use this data to understand everything from how mountains form to why the air is getting smoggy. But there's a big problem: real-world data is messy. Sometimes sensors break, sometimes chemicals are too tiny to see, and sometimes the data just goes missing. When this happens, the usual math tools get confused because they can't handle the "missing" pieces without throwing away the whole smoothie or changing the recipe in a way that makes it unrecognizable.

This paper tackles that exact headache. The authors, Jason Pillay and his team, have invented a new way to do math that doesn't throw away the missing pieces. Instead of forcing the data into a shape it doesn't fit, they built a special "detective" algorithm that can guess what the missing parts might be while it sorts the data into groups. Think of it like a puzzle where some pieces are missing; instead of giving up, the detective uses the shape of the surrounding pieces to figure out exactly where the missing ones belong, all while keeping the puzzle's picture true to the original. They tested this detective on fake data with missing pieces and real data about rocks and air pollution. They found that their method is much better at finding the right groups and picking the right number of categories than the old way of just deleting the messy data. It's a way to make sense of incomplete stories without losing the plot.

The Problem: The "Missing Piece" Puzzle

In science, we often deal with data that describes a whole by its parts. Imagine a rock made of different minerals, or a puff of air filled with different chemicals. The rule is simple: all the parts must add up to 100%. If the rock is 50% quartz, it can only be 50% everything else combined. This is called "compositional data."

The problem arises when the data is incomplete. Maybe a machine failed to measure a specific mineral, or a chemical was too faint to see, so the number is hidden. In the past, scientists had two bad options:

Throw it away: Delete the whole sample because one piece is missing. This wastes a lot of information.
Fake it: Guess the missing number and fill it in before doing the math. This can trick the results and make the groups look wrong.

The authors of this paper wanted to find a third way: a method that stays on the "simplex" (the mathematical name for that 100% rule) and handles the missing pieces directly, without deleting or faking them.

The Solution: A Smart Detective Algorithm

The authors developed a new version of a famous math tool called the Expectation-Maximisation (EM) algorithm. You can think of this algorithm as a super-smart detective trying to sort a messy pile of clues into different groups.

Here is how the detective works in this new method:

The Setup: The detective knows there are different "types" of rocks or air samples (called clusters), but it doesn't know which sample belongs to which type yet.
The Guess (E-step): When the detective sees a sample with missing pieces, it doesn't panic. It looks at the pieces it does see and asks, "If this sample belongs to Group A, what would the missing pieces likely be?" It calculates the probability of the missing parts based on the rules of the group.
The Update (M-step): Using those guesses, the detective updates its understanding of what Group A and Group B actually look like.
The Loop: It repeats this process over and over. With every loop, the guesses get better, and the groups become clearer.

The magic is that this happens all at once. The algorithm figures out the missing values and sorts the samples into groups simultaneously, all while respecting the rule that everything must add up to 100%.

The Test: Simulations and Real-World Mysteries

To see if their detective was any good, the authors ran two types of tests.

1. The Simulation Lab (The Fake Data)
They created 1,000 fake datasets where they knew the true groups but then hid different amounts of data (from 0% to 90% missing).

The Result: Even when 90% of the data was missing, their method could still sort the samples with surprising accuracy.
The Comparison: They compared this to the old method of just deleting the incomplete samples. When data was missing, the old method failed to find the correct number of groups. The new method, however, kept finding the right number of groups (four, in their test) much more often, even when the data was very sparse.
Censoring: They also tested "censored" data, where a value is hidden because it's too small to measure (like a chemical below a detection limit). The new method handled this just as well as the missing data.

2. The Real-World Mysteries
They applied their method to two real datasets to see if it could solve actual scientific puzzles.

Mystery 1: The Earth's Mantle (Xenoliths)
They analyzed 1,256 samples of rocks from deep inside the Earth. These rocks had 33% of their measurements missing because they were collected from different places using different tools.
- The Finding: The algorithm sorted the rocks into four distinct groups. These groups matched real geological types perfectly. For example, one group was rich in magnesium and silicon (peridotites) from ancient stable parts of the Earth's crust, while another had more iron and chromium. The method successfully identified these rock types without needing to throw away the messy samples.
Mystery 2: The Air We Breathe (PM2.5)
They looked at data from the US Air Quality System, tracking tiny particles in the air. This data had both missing values and "censored" values (chemicals too small to measure).
- The Finding: The algorithm found four clear types of air pollution.
  - Group 1: Cold air with lots of sulfates and nitrates (often from industrial emissions).
  - Group 2: A mix of carbon and nitrates.
  - Group 3: Hot air where a huge chunk of the pollution was unknown (the "residual" part was huge), suggesting invisible particles that don't absorb light well.
  - Group 4: Warm air dominated by organic carbon (like smoke or exhaust), which is common in cities.
- The Insight: The method showed that in warmer weather, the pollution composition changes, and a lot of it is made of things the sensors couldn't even see. This gives scientists a new clue about what to look for in the air.

Why This Matters

The paper suggests that we don't have to choose between throwing away bad data or faking numbers. By building a model that understands the rules of the game (the 100% sum) and can handle missing pieces, scientists can get clearer answers from messy data.

The authors found that their method is better at finding the right number of groups and sorting samples correctly than the old "delete the bad data" approach. While the paper doesn't claim this solves every problem in the world, it shows that for rocks and air quality, this new detective is a powerful tool. It allows researchers to see the full picture, even when some of the pieces are hidden in the dark.

Technical Summary: Handling Missingness and Censoring in Dirichlet Mixture Models

Problem Statement
Compositional data, which describe the relative proportions of parts constituting a whole, reside on a simplex rather than unconstrained Euclidean space. A fundamental limitation in analyzing such data is the prevalence of incomplete observations, including missing values (due to equipment malfunction, non-response, or sampling limits) and censored values (where concentrations fall below detection thresholds). Traditional likelihood-based methods for compositional models generally require fully observed compositions. Consequently, analysts often resort to discarding partially observed data (case deletion) or transforming data into unconstrained spaces (e.g., via log-ratio transformations) before analysis. These approaches risk sacrificing interpretability, coherence, and statistical efficiency, particularly because the constant-sum constraint links every component, meaning uncertainty in one part affects the interpretation of all others. Furthermore, standard imputation methods separate the handling of missingness from the subsequent clustering analysis, failing to incorporate imputation uncertainty into parameter estimation.

Methodology
This paper proposes a likelihood-based Expectation–Maximisation (EM) algorithm for fitting finite mixtures of Dirichlet distributions directly on the simplex, accommodating both missing and censored components without leaving the compositional space.

Theoretical Framework: The approach utilizes the properties of the Dirichlet distribution and the truncated Dirichlet distribution. The random vector is partitioned into observed ( $X_o$ ) and unobserved ( $X_u$ ) components. The method derives the marginal density of the observed components and the conditional distribution of the unobserved components given the observed ones and the cluster membership. The unobserved components are modeled as following a truncated Dirichlet distribution within specific bounds defined by the censoring mechanism or missingness pattern.
The EM Algorithm:
- E-step: Computes the expected value of the complete log-likelihood. This involves calculating the posterior probability of cluster membership ( $\tilde{z}_{ig}$ ) for each observation, which accounts for the probability of the unobserved part falling within its specific truncation region. It also computes the expected values of the log of the unobserved components ( $\tilde{E}_{ikg}$ ), derived using the properties of the truncated Dirichlet distribution and the digamma function.
- M-step: Maximizes the expected log-likelihood with respect to the mixing proportions ( $\pi_g$ ) and the Dirichlet parameters ( $\alpha_g$ ). The mixing proportions are updated via standard weighted averages. The Dirichlet parameters are reparameterized using unconstrained variables ( $\beta_{kg} = \ln \alpha_{kg}$ ) and optimized using a Newton–Raphson (NR) algorithm.
Handling Mechanisms: The framework operates under a Coarsening At Random (CAR) assumption, which encompasses Missing Completely At Random (MCAR), Missing At Random (MAR), and deterministic censoring. The algorithm requires only a single observed component per composition to function, allowing it to handle datasets with no fully observed rows.
Initialization and Convergence: The algorithm employs method-of-moments estimates for initialization and uses the Aitken acceleration criterion to assess convergence of the observed-data log-likelihood.

Key Contributions and Results

Simultaneous Estimation and Imputation: The proposed method performs parameter estimation, cluster allocation, and model-based imputation simultaneously within a single likelihood-based procedure, preserving the compositional structure.
Simulation Study: Extensive simulations were conducted on datasets generated from mixtures of four Dirichlet distributions with varying dimensions ( $p=6$ $p = 6$ to $10$) and levels of incompleteness (up to 90% missing or censored).
- Clustering Performance: The algorithm maintained non-zero clustering accuracy and Adjusted Rand Index (ARI) even at high levels of missingness (90%), though performance naturally declined as data sparsity increased.
- Model Selection: The proposed method demonstrated a significantly higher probability of correctly identifying the true number of clusters (four) using AIC, BIC, and ICL criteria compared to the alternative of case deletion. Case deletion resulted in a rapid drop to zero probability of correct model selection once missingness exceeded 50%, whereas the proposed method remained robust.
- Censoring vs. MAR: The algorithm performed effectively under both MAR and censoring mechanisms, though model selection accuracy decayed slightly faster under censoring due to the harsher nature of the coarsening.
Real-World Applications:
- EarthChem Mantle Xenoliths: Applied to a geochemical dataset ( $n=1256$ , 33% missing) to identify rock types and speciation methods. A four-component mixture revealed interpretable profiles, distinguishing peridotites from cratonic settings, pyroxenites from intraplate settings, and other distinct geochemical groups, with results aligning with geological descriptors not used in the clustering.
- Air Quality System (AQS) PM2.5: Applied to particulate matter speciation data ( $n=99$ ) containing both missing and left-censored values. A four-component model characterized distinct pollution profiles, differentiating between clusters dominated by secondary inorganic ions (associated with cooler temperatures), mixed carbon/inorganic sources, and organic-dominated compositions (associated with warmer temperatures).

Significance and Claims
The paper claims that the proposed methodology addresses a critical gap in compositional data analysis by enabling likelihood-based inference and clustering directly on the simplex in the presence of unobserved parts. By avoiding data deletion or transformation to unconstrained spaces, the method preserves the interpretability of the original variables and the geometric constraints of the data. The authors assert that their approach substantially improves model selection and clustering performance relative to existing alternatives, particularly in scenarios with severe data incompleteness. The work reinforces the utility of Dirichlet mixture models for heterogeneous compositional populations and suggests a shift from adapting data to fit unconstrained models toward developing estimation tools that respect the geometry of compositional data.

The Problem: The "Missing Piece" Puzzle

The Solution: A Smart Detective Algorithm

The Test: Simulations and Real-World Mysteries

Why This Matters

Technical Summary: Handling Missingness and Censoring in Dirichlet Mixture Models

More like this