Technical Summary: Learning from Uncertainty-Dependent Missing Labels for Semi-Supervised Classification

1. Problem Statement

In semi-supervised classification, missing labels are traditionally viewed as a source of information loss, reducing efficiency and complicating inference. However, in many practical scenarios—such as medical imaging, active learning, and online moderation—labels are not missing at random (MAR) in a passive sense. Instead, the probability of a label being missing often depends on the observed features ($Y$) and, crucially, on the posterior classification uncertainty derived from the label model itself.

The central problem addressed is how to characterize the information content of such uncertainty-dependent missing labels. While classical information theory dictates that observing a reduced version of a full experiment cannot increase information beyond the augmented full-data experiment, this paper investigates whether the missingness indicator ($M$) itself, when generated by an uncertainty-dependent mechanism, can act as an observable signal that improves estimation and classification performance relative to standard fully labeled or non-informative partially labeled baselines under a fixed budget.

2. Methodology

2.1. Statistical Framework

The authors consider a classification setting with $g$ classes, features $Y$, and labels $Z$. The missingness indicator $M \in \{0, 1\}$ denotes whether a label is observed ($M=0$) or missing ($M=1$). The missingness mechanism is modeled as:
$$r_{\theta,\xi}(Y) = \Pr(M=1 \mid Y; \theta, \xi) = h\{\eta_{\theta,\xi}(Y)\}$$
where $h$ is an inverse link function, and $\eta_{\theta,\xi}(Y)$ is a predictor depending on the label model parameters $\theta$, mechanism parameters $\xi$, and a summary of posterior classification uncertainty $u(Y; \theta)$. Examples of $u(Y; \theta)$ include posterior Shannon entropy, negative log-entropy, or posterior variance.

The observed-data likelihood is constructed as:
$$L(\theta, \xi) = \prod_{j=1}^n \{p_\theta(Y_j, Z_j)[1 - r_{\theta,\xi}(Y_j)]\}^{1-M_j} \{p_\theta(Y_j)r_{\theta,\xi}(Y_j)\}^{M_j}$$
where the first term applies when labels are observed and the second (marginal feature density) when they are missing.

2.2. Information Decomposition

The core methodological contribution is a likelihood-based information theory that decomposes the observed Fisher information $I_{obs}(\theta)$.

Correctly Specified Case:
Using Louis's observed-information identity, the authors derive a decomposition separating the information into a partial-labeling component and a mechanism-curvature term:
$$I_{obs}(\theta) = \underbrace{I_{CC}(\theta) - I_{CC}^{(miss,r)}(\theta, \xi)}_{\text{Partial-labeling component}} + \underbrace{I_{mech}(\theta, \xi)}_{\text{Mechanism curvature}}$$

• $I_{CC}(\theta)$: Complete-data Fisher information.
• $I_{CC}^{(miss,r)}(\theta, \xi)$: Weighted missing-information loss.
• $I_{mech}(\theta, \xi)$: A positive semidefinite term quantifying the curvature contributed by the observed missingness indicators $M_j$. For logistic links, this term depends on the variance of the missingness probability and the gradient of the uncertainty summary.

Misspecified Case:
Recognizing that both the label model and the mechanism may be misspecified in practice, the authors extend the framework to the Godambe–Eicker–Huber–White (sandwich) covariance framework. They derive the sensitivity and sandwich-covariance partitions for joint estimation of $(\theta, \xi)$, showing that the structural decomposition (partial-labeling + mechanism curvature) persists even under misspecification, though uncertainty quantification shifts from inverse Fisher information to the sandwich covariance.

2.3. Theoretical Bounds

The paper clarifies the relationship between the observed partially labeled experiment and the "augmented" experiment where both labels and mechanism indicators are observed. It proves that while uncertainty-dependent missingness can yield favorable missingness (greater information than a budget-matched non-informative baseline), it cannot exceed the information bound of the augmented experiment $(Y, Z, M)$. The observed data $(Y, M, Z_{obs})$ are a coarsening of the augmented data, satisfying standard information inequalities.

2.4. Risk Analysis

For plug-in classifiers, the authors connect the information decomposition to margin-based excess-risk bounds. In regular two-component mixture settings, they establish that the excess risk converges at the parametric rate $O_p(n^{-1})$. The constants in this rate are determined by the nuisance-adjusted information in discriminant directions, implying that a favorable mechanism curvature can reduce the asymptotic variance of the decision boundary.

3. Key Contributions

1. Information Decomposition: Derivation of a Fisher-information decomposition that explicitly isolates a nonnegative "mechanism-curvature" term. This term explains how the missingness indicator, when dependent on posterior uncertainty, carries additional information about the classifier.
2. Robustness under Misspecification: Extension of the decomposition to the sandwich covariance framework under joint misspecification of the label model and the missingness mechanism, providing a rigorous basis for inference in practical settings where working models are used.
3. Excess-Risk Rates: Establishment of margin-based excess-risk bounds for plug-in classifiers under uncertainty-dependent missingness, demonstrating that favorable missingness leads to improved classification performance (lower asymptotic variance) under fixed labeling budgets.
4. Clarification of "Favorable Missingness": A rigorous definition and proof that favorable missingness is a relative gain over ordinary fully labeled or budget-matched non-informative baselines, not a violation of classical information inequalities regarding the augmented full-data experiment.

4. Results

4.1. Numerical Illustrations (Gaussian Mixtures)

• Information Gain: In a two-component Gaussian mixture setting, Monte Carlo simulations demonstrate that entropy-dependent missingness mechanisms can yield a three-fold increase in Fisher information along the discriminant direction compared to a non-informative (MCAR) baseline with the same missingness rate.
• Regime Dependence: The information gain is non-monotonic with respect to the missingness rate and design sensitivity. It is maximized when there is moderate class overlap, a non-excessive missingness rate, and a mechanism sensitive enough to concentrate missingness near high-uncertainty observations without saturating.
• Cost-Benefit Analysis: Under a fixed total budget (balancing feature collection cost and labeling cost), the optimal missingness rate increases with the relative cost of labeling. Uncertainty-dependent designs allow for larger feature sample sizes while maintaining or improving estimation efficiency compared to fully supervised designs.

4.2. Case Study (Medical Diagnosis)

The framework was applied to a gastrointestinal dataset (colonoscopy videos) where labels were derived from endoscopist consensus.

• Mechanism Validation: The data confirmed that label missingness (lack of consensus) was strongly associated with higher posterior entropy (uncertainty).
• Performance: Entropy-based semi-supervised models (SSL) outperformed a supervised benchmark trained only on the 35 available labeled cases.
  • The SSLlogit model achieved the lowest prediction error rate (0.1325) compared to the supervised benchmark (0.1775).
  • The SSL models exhibited larger fitted Fisher information matrices, indicating higher estimation efficiency driven by the mechanism-curvature term.

5. Significance and Claims

The paper claims to provide a likelihood-based framework for understanding how uncertainty-dependent missingness reshapes the information geometry of semi-supervised learning. Its significance lies in:

• Reframing Missingness: Moving the perspective of missing labels from a nuisance to be imputed or ignored, to a structured signal that can be explicitly modeled to improve inference.
• Resolving the Paradox: Clarifying that "favorable missingness" does not violate classical information theory; rather, it exploits the fact that the missingness indicator $M$ is an observable variable generated by a mechanism linked to the classifier's uncertainty.
• Practical Utility: Demonstrating that explicitly modeling the missingness mechanism (e.g., via entropy-dependent masking) can lead to tangible improvements in both parameter estimation (via increased Fisher information) and classification accuracy (via reduced excess risk) under fixed resource constraints.
• Robustness: Providing a theoretical foundation (via the sandwich estimator) that remains valid even when the working models for the label distribution or the missingness mechanism are imperfect, which is typical in real-world applications.

The authors conclude that while the gains are local and regime-dependent, the framework offers a principled way to leverage selection effects in data collection for more efficient semi-supervised learning.