📊 statistics

Clustering data with values missing at random using scale mixtures of multivariate skew-normal distributions

This paper proposes an augmented EM-type algorithm for model-based clustering of data with missing values at random by extending the finite mixture of scale mixtures of multivariate skew-normal distributions to simultaneously handle skewness, heavy tails, and incomplete observations.

Original authors: Jason Pillay, Cristina Tortora, Antonio Punzo, Andriette Bekker

Published 2026-07-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jason Pillay, Cristina Tortora, Antonio Punzo, Andriette Bekker

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery by grouping suspects based on their habits. In the world of data science, this is called "clustering." Usually, detectives have a perfect lineup where every suspect shows up with a full file of information. But in the real world, files are often missing pages, ink is smudged, or witnesses forget details. This is "missing data." If you try to solve the mystery by ignoring the missing files, you might miss the most important clues. If you try to guess what's missing, you might accidentally invent a suspect who never existed.

To make sense of messy, incomplete information, scientists often use a tool called a "mixture model." Think of this like a bag of different colored marbles. If you reach in and pull one out, you don't know which color it is, but you know the bag is a mix of red, blue, and green. The goal is to figure out how many colors are in the bag and what the "average" red marble looks like. For a long time, scientists assumed these marbles were perfectly round and symmetrical, like standard dice. But real-world data is often lopsided—some marbles are stretched out, or have heavy tails where extreme values hang out. To handle this, scientists developed "skew-normal" distributions, which are like stretchy, lopsided marbles that can bend to fit weird shapes.

However, there was a problem: these fancy, stretchy marbles were great when the data was complete, but they fell apart when the files had missing pages. You couldn't just ignore the missing parts, and guessing them was risky. This paper steps in to fix that gap. It takes the powerful, stretchy "skew-normal" marbles and teaches them how to handle missing information without breaking the rules of the game.

The authors, a team of statisticians from South Africa, the USA, and Italy, have built a new mathematical engine called the "Finite Mixture of Scale Mixtures of Multivariate Skew-Normal" (FMSMSN) family. Imagine this family as a toolbox containing four different types of stretchy, lopsided marbles: the standard skew-normal, the skew-t (which handles extreme outliers), the skew-slash (which handles even wilder outliers), and the skew-variance-gamma (the most flexible of all).

The big breakthrough in this paper is that they figured out how to use this entire toolbox even when the data is incomplete. They did this by assuming the missing data is "Missing At Random" (MAR). In detective terms, this means the reason a file is missing isn't because the suspect is hiding something specific about themselves, but perhaps because the file got lost in the mail or the witness was busy. The missingness doesn't depend on the secret value itself. Under this assumption, the authors derived a new set of rules (a modified algorithm) that allows the computer to "fill in the blanks" mathematically while it figures out the groups, rather than just guessing or throwing away the data.

To see if their new engine worked, the team ran a series of computer simulations. They created fake datasets with two groups of data, some of which were close together and some far apart. They then deliberately erased 0%, 20%, 40%, 60%, and even 80% of the information at random. They tested all four types of marbles from their toolbox to see which one could still find the groups correctly and guess the right shapes of the marbles.

The results showed that as more data went missing, it became harder for all the models to find the groups, which is expected. However, the more complex and flexible models (like the skew-variance-gamma) generally did a better job of recovering the true shapes of the data, even with up to 80% of the information missing. The team found that while no model was perfect when data was missing, their new method was far superior to simply deleting the incomplete rows, which would have left them with almost no data to work with.

Finally, the researchers took their new method out of the simulation lab and applied it to real-world data: global carbon dioxide (CO2) emissions. They looked at emissions from seven different sectors (like electricity, manufacturing, and transport) for countries around the world. The catch? Over 84% of the rows in this dataset were incomplete. If they had used the old "delete the missing" method, they would have had to throw away almost the entire dataset. Instead, they used their new algorithm.

The algorithm successfully grouped the countries into two distinct clusters. One group included nations with generally lower emissions, such as parts of North Africa, Canada, and Japan. The other group included countries with higher emissions, including the US, China, India, and many nations in the Global South. The analysis suggested a link between improving economic growth (GDP) and increased carbon emissions, highlighting that developing economies often face a trade-off between growth and environmental impact. The study concludes that by using this flexible, missing-data-friendly approach, scientists can now uncover patterns in messy, real-world data that were previously impossible to see, offering a clearer picture of global trends without having to discard valuable information.

Technical Summary: Clustering Data with Values Missing at Random Using Scale Mixtures of Multivariate Skew-Normal Distributions

Problem Statement
Model-based clustering is a standard tool for analyzing high-dimensional data, yet it faces significant challenges when datasets exhibit skewness, heavy tails, and missing values. While Gaussian Mixture Models (GMMs) are computationally efficient, they often fail to capture asymmetric cluster geometries, leading practitioners to overestimate the number of clusters to compensate for asymmetry. Although the Finite Mixture of Scale Mixtures of Multivariate Skew-Normal (FMSMSN) family has been developed to address skewness and excess kurtosis, existing literature lacks a unified framework to apply these models to incomplete data. Current approaches often rely on listwise deletion, which discards valuable information and introduces bias, or computationally expensive imputation techniques. Furthermore, previous extensions of the FMSMSN family to handle missing data have been limited to specific members of the family (e.g., only skew-t or skew-normal) rather than the general family.

Methodology
The paper proposes a general framework for fitting FMSMSN models to data with Missing At Random (MAR) values. The methodology is built upon three theoretical and algorithmic pillars:

Distributional Theory under MAR: The authors extend the distribution theory of the multivariate skew-normal (SN) distribution to explicitly handle random vectors with observed ( $X^{(o)}$ ) and missing ( $X^{(m)}$ ) components. They derive the full probability distribution governing the missing components under the MAR assumption. This involves partitioning the location vector ( $\mu$ ), scale matrix ( $\Sigma$ ), and skewness vector ( $\lambda$ ) into observed and missing sub-components. The authors establish that the conditional distribution of the missing components, given the observed ones, follows a specific skew-normal form with updated parameters.
Stochastic Representation and Moments: Utilizing the stochastic representation of the FMSMSN family, the paper derives closed-form expressions for the conditional expectations and covariances of the missing data. These derivations are crucial for the Expectation-Maximisation (EM) algorithm, as they allow for the calculation of the necessary sufficient statistics without resorting to numerical integration.
ECM Algorithm: The authors develop an Expectation-Conditional Maximisation (ECM) algorithm tailored for incomplete observations.
- E-step: Computes the conditional expectations of the latent variables (allocation vectors, scaling variables, and truncated normal variables) and the missing data components. The paper provides closed-form expressions for these expectations, including the cross-products of missing values.
- M-step: Updates the model parameters (mixing proportions, location, scale, skewness, and hyperparameters) by maximizing the expected complete-data log-likelihood. The M-step is divided into conditional maximisation steps to handle the interdependence of parameters.
- Convergence: The algorithm employs the Aitken acceleration criterion to determine convergence, ensuring the observed log-likelihood stabilizes.

Key Contributions
The paper makes two primary contributions:

Theoretical Extension: It provides the first derivation of the full probability distribution for missing components within the entire FMSMSN family under the MAR mechanism. This includes establishing crucial closed-form expressions for the expectation and covariance structures, allowing the family to be applied generally rather than to isolated special cases.
Methodological Innovation: It proposes a generalized estimation procedure (ECM algorithm) that simultaneously handles incomplete observations and latent variables for the FMSMSN family. A distinct advantage of this approach is that it provides a set of imputed values for the missing data at the end of the algorithm with minimal additional computational cost, preserving the underlying distributional structure.

Results
The proposed method was evaluated through extensive simulation experiments and a real-world application:

Simulation Studies: The authors simulated bivariate data from two-component mixtures using four specific distributions: Multivariate Skew-Normal (SN), Skew-Slash (SS), Skew-t (ST), and Skew-Variance-Gamma (SVG). They tested performance across varying levels of cluster overlap (well-separated vs. close) and missing data proportions (0% to 80%).
- Clustering Performance: Measured by the Adjusted Rand Index (ARI), results showed an inverse relationship between the proportion of missing data and clustering accuracy. However, the FMSMSN family maintained robust performance even with high missingness (up to 80%). The more flexible distributions (SVG and SS) generally outperformed the simpler SN when the data was generated from complex distributions, particularly in scenarios with close cluster proximity.
- Parameter Recovery: Measured by Absolute Bias (AB) and Root Mean Squared Error (RMSE), the SVG distribution consistently demonstrated the lowest error across varying missingness levels and cluster proximities. The recovery of skewness parameters ( $\lambda$ ) showed unique behavior, deviating from the general trend under close cluster proximity.
Real-World Application: The method was applied to global CO2 emissions data from the EDGAR database (2022), which contained 84.13% incomplete rows across seven economic sectors.
- Model selection via Bayesian Information Criterion (BIC) favored a two-component Skew-Variance-Gamma mixture.
- The analysis successfully grouped countries into two distinct clusters: one representing countries with lower overall emissions (including parts of Europe, North Africa, and developed economies like Japan and Canada) and another representing countries with higher emissions (including the Global South, the US, UK, Australia, China, and India).
- The results highlighted a link between improving GDP and increased carbon emissions, as well as distinct driving factors between the Global North and South. The ability to model the full dataset without listwise deletion was critical, as the complete subset was too small for meaningful analysis.

Significance and Claims
The paper claims that the proposed framework addresses a critical gap in the literature by enabling the application of the entire FMSMSN family to incomplete data. The significance lies in the ability to:

Handle data exhibiting both skewness and heavy tails without discarding incomplete observations.
Provide a flexible modeling toolkit where practitioners can select the most appropriate distribution (from SN to SVG) based on the data's characteristics, balancing parsimony and complexity.
Offer a computationally efficient alternative to multiple imputation, providing both parameter estimates and a heuristic reconstruction of the incomplete dataset.

The authors conclude that while performance is naturally lower than in complete datasets, the method yields meaningful results that would otherwise be inaccessible. They note that future work will focus on extending this methodology to censored data, of which MAR is a special case.

Technical Summary: Clustering Data with Values Missing at Random Using Scale Mixtures of Multivariate Skew-Normal Distributions

More like this