📊 statistics

Informative Label Missingness in Multiclass Classification: Information Geometry and Excess Risk

This paper develops a likelihood-based theory demonstrating that in multiclass classification, informative label missingness can reduce asymptotic excess risk by aligning information gains from the missingness pattern with decision boundary perturbations, even when complete classification retains higher global Fisher information.

Original authors: Fariborz Setoudehtazangi, Geoffrey McLachlan

Published 2026-09-23

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Fariborz Setoudehtazangi, Geoffrey McLachlan

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of machine learning, computers learn to recognize patterns by studying examples. Imagine a teacher showing a student thousands of pictures of cats and dogs, telling them which is which. The student learns to spot the features that distinguish one from the other. This process, known as classification, relies heavily on having correct labels for every single image. However, in the real world, data is often messy. Sometimes, a teacher might forget to label a picture, or a human expert might decide an image is too confusing to classify and leave it blank. For decades, statisticians have operated under a simple rule: having a label is always better than not having one. If you have a picture of a cat but don't know it's a cat, that picture is considered less useful than one where the label is clearly written. The missing information was seen as a pure loss, a gap in the data that made the student's job harder.

But what if the reason a label is missing actually tells you something important? What if the expert only skipped labeling the pictures that were hardest to distinguish? In those cases, the fact that a label is missing is not just a gap; it is a clue. It signals that the image sits right on the blurry line between two categories. This subtle possibility—that the pattern of missing data itself carries information—has been a subject of growing interest, but it remained difficult to understand in complex situations involving more than two categories. A new study by researchers Fariborz Setoudehtazangi and Geoffrey McLachlan explores this phenomenon, moving beyond simple comparisons to map out exactly when and why missing labels can sometimes help a computer learn better than if all labels were present.

The researchers focused on a scenario where a computer is trying to sort data into three or more groups, a task that is far more complex than a simple choice between two options. They developed a mathematical framework to track how information flows when labels are missing in a specific, non-random way. Their work reveals that the value of a dataset is not just about how many labels you have, but where those missing labels are located. If the missing labels cluster around the most difficult decision points—the areas where the computer is most confused—the pattern of absence can actually guide the computer toward a more accurate understanding of the boundaries between groups.

To test this, the team constructed a detailed simulation using a three-class system where the groups were defined by curved, complex boundaries rather than simple straight lines. They created a scenario where the probability of a label being missing depended entirely on how uncertain the computer was about that specific data point. In their reference setup, they removed thirty percent of the labels. When these missing labels were distributed randomly, the computer's performance dropped, as expected. However, when the missing labels were concentrated in the regions of highest uncertainty, the computer's performance improved. In fact, the model trained on this partially labeled, informative data made fewer mistakes than the model trained on the complete set of data where every label was known.

This counterintuitive result happens because the missingness mechanism acts as a secondary source of information. When the computer sees that a label is missing in a specific spot, it learns that this spot is a place of high confusion between the classes. This extra signal helps the computer refine the shape of the decision boundary more precisely than it could by looking at the labeled data alone. The researchers showed that this advantage does not require the missing data to provide more total information than the complete data. Instead, it depends on the alignment of that information. The missing labels must provide clues in the specific directions that matter most for drawing the lines between the groups. If the missing labels are scattered in directions that do not affect the decision boundary, they offer no help.

The study also examined how this advantage emerges as the missingness pattern becomes more dependent on uncertainty. They found that the benefit is not immediate. If the connection between missing labels and uncertainty is weak, the loss of the actual labels hurts the computer more than the new clues help it. The system initially gets worse. Only when the dependence becomes strong enough does the helpful signal from the missingness pattern overcome the cost of the missing labels. This creates a tipping point, a specific threshold of informativeness where the strategy flips from being harmful to being beneficial. The researchers mapped out these thresholds, showing that the exact point where the switch happens depends on how separated the groups are and how balanced the data is.

To ensure these findings were not just theoretical, the team ran extensive computer experiments with finite amounts of data, mimicking real-world sample sizes. They confirmed that the theoretical predictions held true even with limited data. The models trained on the informative missing data consistently outperformed those trained on random missing data and, in the specific conditions they tested, even outperformed the models trained on the full dataset. They also applied their method to a real-world dataset involving medical measurements of the human spine, where patients were classified into three categories: normal, spondylolisthesis, or disk hernia. By artificially removing labels based on uncertainty and then modeling that removal process, they found that explicitly accounting for the missingness mechanism improved the model's ability to predict probabilities, even if the final error rate did not always drop below that of the complete data.

The work challenges the long-held assumption that more labeled data is always better. It suggests that in situations where experts or systems selectively omit labels from the most ambiguous cases, ignoring that selection process is a mistake. By treating the pattern of missingness as a source of data rather than just a nuisance, statisticians can build better classifiers. The researchers emphasize that this is not a universal rule; it is a regime-dependent phenomenon. It works only when the missingness is informative and strong enough to compensate for the lost labels. If the missingness is random or weak, the traditional approach of discarding incomplete data remains the best path.

Ultimately, this study provides a new way to look at the geometry of learning. It shows that the value of information is not uniform; it depends on where that information sits relative to the decision boundaries. A missing label in a clear, easy region is a waste, but a missing label in a confusing region is a signpost. By understanding this alignment, researchers can better design learning systems that know how to read the silence between the data points, turning gaps in the record into guides for the future.

Technical Summary: Informative Label Missingness in Multiclass Classification: Information Geometry and Excess Risk

Problem Statement
In multiclass classification, partially labelled data arises when feature vectors are observed but class memberships for some observations are missing. While standard semi-supervised learning often assumes labels are Missing Completely at Random (MCAR), many practical scenarios involve informative missingness, where the probability of a label being missing depends on the observed features and the classification model parameters. A counterintuitive phenomenon, previously identified in binary homoscedastic Gaussian settings, suggests that under specific informative missingness mechanisms, a partially labelled sample can yield a lower asymptotic classification risk than a completely labelled sample. However, the theoretical understanding of this phenomenon in general multiclass problems remains limited. Existing results rely heavily on binary structures, failing to account for the complex geometry of multiclass Bayes boundaries, which involve active pairwise faces and higher-order junctions. Furthermore, global improvements in Fisher information do not necessarily translate to improved classification performance, as risk depends on how estimation errors perturb the decision boundary.

Methodology
The authors develop a general likelihood-based theory for informative label missingness in parametric multiclass classification, linking information geometry to classification risk.

Efficient Information Decomposition:
The paper derives a decomposition of the efficient Fisher information for the classification parameters ( $\theta$ ) after eliminating nuisance parameters specific to the missingness mechanism ( $\xi$ ). The effective information under partial classification ( $I_{PC}^{eff}$ ) is expressed as:
$I_{PC}^{eff} = I_{CC} - E[q(Y)I_{Z|Y}(Y)] + I_{M}^{eff}$
where $I_{CC}$ is the information from a complete sample, the second term represents information lost due to unavailable labels (weighted by the missingness probability $q(Y)$ ), and $I_{M}^{eff}$ is the efficient information contributed by the missingness indicators themselves. This decomposition highlights that the cost of missing labels depends on where in the feature space they occur, not just the marginal proportion.
Excess Risk Geometry:
The authors derive a local quadratic expansion for the excess classification risk of a plug-in Bayes classifier. They show that the leading term of the risk is determined by the curvature of the active pairwise Bayes faces (surfaces where two classes are tied for the maximum posterior probability and strictly exceed all others). Triple and higher-order junctions contribute only to higher-order terms under regularity and transversality conditions. The excess risk is given by:
$E[R(\hat{\theta})] - R^* = \frac{1}{2n} \text{tr}(H_R V) + o(n^{-1})$
where $V$ is the asymptotic covariance matrix of the estimator and $H_R$ is a curvature matrix induced by the active faces.
Classification-Weighted Comparison:
By combining the information and risk expansions, the paper establishes a criterion for comparing complete and partial classification. The difference in leading excess risk depends on the trace of the product of the curvature matrix and the difference in inverse information matrices:
$\Delta R = \text{tr}[H_R (A^{-1} - J^{-1})]$
where $A$ and $J$ are the information matrices for complete and partial classification, respectively. Using a generalized eigenvalue decomposition, the authors show that $\Delta R > 0$ (favoring partial classification) occurs when information gains are concentrated in directions that significantly perturb the active decision boundaries, even if the partial classification does not globally dominate the complete classification in terms of Fisher information ( $J \not\succeq A$ ).
Local Analysis near MCAR:
The paper analyzes the transition from MCAR to informative missingness. It demonstrates that redistributing missing labels based on classification uncertainty alters the loss of class-label information at the first order, while the information gain from the missingness pattern appears only at the second order. Consequently, weak informative missingness may initially worsen classification performance before becoming beneficial as the dependence strengthens.

Key Results

Directional Alignment: Favorable informative missingness is a directional alignment phenomenon. A partially classified sample can outperform a complete one if the information gained from the missingness pattern aligns with parameter directions that are critical for the decision boundary, even if information is lost in other directions.
Phase Transitions: In a three-class Gaussian Quadratic Discriminant Analysis (QDA) model with unequal covariances, the authors identify a "phase transition." As the strength of the dependence between missingness and classification uncertainty increases (while holding the marginal missing rate fixed), the system moves from an unfavourable regime (where partial classification is worse) to a favourable one.
Geometry Dependence: The magnitude and sign of the classification advantage are sensitive to the underlying geometry. Severe class overlap can negate the benefit, and strong prior imbalance can reverse it. However, the advantage persists across a range of covariance heterogeneities.
Finite-Sample Validation: Monte Carlo experiments confirm that the asymptotic predictions hold in finite samples. The empirical excess risks and covariance matrices align with the theoretical limits, and the ordering of performance (Informative Partial Classification < Complete Classification < MCAR) is reproduced.
Real-Data Application: A semi-synthetic application to the Vertebral Column dataset (three classes: disk hernia, spondylolisthesis, normal) illustrates that explicitly modelling the informative missingness mechanism improves probabilistic prediction (log loss, Brier score) compared to ignoring the mechanism, though the improvement in 0-1 error is modest and regime-dependent.

Significance and Claims
The paper claims to provide a general framework that explains how and when informative label missingness translates into classification risk improvements. Its primary contribution is not the discovery that informative missingness can be beneficial (a phenomenon already known in binary settings), but the development of a multiclass theory that:

Separates the information lost from labels and the information gained from the missingness pattern.
Links these information changes to the specific geometry of the active Bayes decision boundary.
Demonstrates that global Fisher information dominance is not a necessary condition for classification improvement; rather, the alignment of information gains with boundary-relevant directions is the critical factor.
Clarifies that the benefit is not universal but depends on the regime (strength of dependence, class separation, prior imbalance).

The authors conclude that the statistical value of a partially labelled sample cannot be determined by the proportion of observed labels or unweighted Fisher information comparisons alone. Instead, it requires an analysis of how information gains and losses align with the geometry of the classification problem.

More like this