When to Trust Confidence Thresholding: Calibration Diagnostics for Pseudo-Labelled Regression

Technical Summary: Calibration Diagnostics for Pseudo-Labelled Regression

1. Problem Statement

Modern machine learning pipelines increasingly utilize calibrated probability scores from classifiers trained on small labelled subsets ($L$) to inform downstream regression tasks on large unlabelled populations ($U$). Common applications include estimating population-level effects, prevalences, or disparities for latent groups (e.g., protected attributes in fairness audits or disease risk in medical imaging).

A dominant practice in this semi-supervised setting is pseudo-labelling: converting the calibrated probability $p$ into a hard binary label $\hat{G}$ by applying a confidence threshold $\tau_{thr}$ (e.g., 0.9). While empirically successful for classification, this paper addresses a critical gap: how does this thresholding bias the downstream regression coefficient? Existing literature lacks a quantitative account of how confidence thresholding attenuates or distorts the estimand of interest, specifically in the context of partially linear regression models where the goal is to estimate the association between a latent group indicator $G$ and an outcome $Y$, controlling for covariates $X$.

2. Methodology and Setup

The paper operates within a framework where:

• Data: Two i.i.d. samples are observed: a labelled set $L = \{(W_i, X_i, G_i, Y_i)\}$ and a larger unlabelled set $U = \{(W_i, X_i, Y_i)\}$.
• Features: The classifier is trained on a feature set $W$, while the downstream regression controls for a subset $X \subseteq W$. Crucially, the paper enforces a structural separation where $W$ must be strictly richer than $X$ ($X \subsetneq W$) to ensure identifiability.
• Model: The target parameter $\tau$ is defined via a partially linear regression: $E[Y | G, X] = \mu(X) + \tau G$.
• Assumption: The classifier output $p = f(W)$ is conditionally calibrated, satisfying $E[G | p, X] = p$.

Building on a companion identification result [8], the authors utilize a moment equation to identify $\tau$ using the calibrated score $p$ directly (the "soft" estimator). They then analyze the properties of the "hard" estimator, which replaces $p$ with a thresholded binary label $\hat{G} = \mathbb{1}\{p > \tau_{thr}\}$.

3. Key Contributions and Theoretical Results

The paper derives five specific predictions and a diagnostic procedure, formalized as follows:

I. Closed-Form Attenuation Bias (Prediction 1)
The paper derives a closed-form expression for the attenuation bias induced by hard pseudo-labelling. The probability limit of the hard estimator $\hat{\tau}_{hard}$ is:
$$\text{plim } \hat{\tau}_{hard} = \kappa(\tau_{thr}) \cdot \tau$$
where the attenuation factor $\kappa(\tau_{thr})$ is the population OLS slope of the residualized hard label on the residualized soft score.

• Significance: This factor is computable before running any inference, solely from the joint distribution of $(p, X)$ on the unlabelled set. It quantifies exactly how much signal is lost when binarizing the probability at a specific threshold.

II. Sandwich Variance for Soft Estimators (Prediction 2)
The asymptotic variance of the soft estimator (using $p$ directly) scales inversely with the square of the residual score variance $V^* = E[\text{Var}(p|X)]$.

• Implication: Even modest reductions in the informativeness of the score (low $V^*$) lead to massive precision losses. The paper provides a sandwich-variance plug-in that yields valid Wald intervals.

III. Bias-Variance Regret of Confidence Thresholding (Prediction 3)
For methods that restrict pseudo-labelling to a confident subset (e.g., FixMatch-style), the Mean Squared Error (MSE) decomposes into a squared bias term (driven by $\kappa_{FM}$) and a variance term (inflated by the reduced sample size $|C|$).

• Finding: As the threshold becomes more selective, the variance grows monotonically while the bias may fluctuate. The total MSE often becomes variance-driven, making the soft estimator superior in many regimes despite its lack of "hard" labels.

IV. Sensitivity Bound under Calibration Drift (Prediction 4)
Under bounded distribution shift where the calibration condition holds with an error $\eta(p, X)$ bounded by $\delta$, the asymptotic bias is bounded by:
$$|\text{plim } \hat{\tau} - \tau| \leq |\tau| \frac{\delta E|2p-1|}{2V^*}$$

• Implication: Higher residual variance $V^*$ (more informative scores) proportionally reduces the worst-case bias for a given level of calibration drift.

V. The $V^*$-Collapse Boundary (Prediction 5)
The paper identifies a sharp boundary: $V^* = 0$ if and only if $p$ is a deterministic function of $X$.

• Structural Requirement: If the classifier features $W$ are identical to the downstream controls $X$, then $p = f(X)$, $V^*=0$, and the moment equation becomes uninformative (identification fails). The remedy is to ensure $W$ contains features not controlled for in the downstream regression.

4. The Diagnostic Procedure

The authors propose a practical, one-page diagnostic algorithm (Algorithm 1) for practitioners to decide whether confidence thresholding is safe:

1. Compute $V^*$: Estimate the residual variance of the calibrated score after regressing out $X$ on the unlabelled set.
2. Compute $\hat{\kappa}$: For a grid of thresholds, estimate the attenuation factor.
3. Decision Rule:
   • If the implied standard error of the soft estimator is too high (due to low $V^*$), prefer the supervised baseline (using only $L$).
   • If $\hat{\kappa}$ is far from 1 for all thresholds, hard thresholding causes significant attenuation; prefer the soft estimator.
   • Otherwise, either is acceptable, though the soft estimator is recommended for its robustness to threshold choice.

5. Experimental Results

The paper validates these predictions through five controlled simulations and a real-data illustration on the UCI Adult Income dataset:

• Attenuation: Empirical estimates of $\hat{\tau}_{hard}$ matched the closed-form $\kappa(\tau_{thr})$ predictions across six different thresholds with high statistical significance.
• Variance: The sandwich variance formula accurately predicted Monte Carlo variance and achieved nominal 95% coverage.
• Regret: Experiments confirmed that high-confidence thresholds often lead to high MSE due to variance inflation, validating the bias-variance trade-off analysis.
• Sensitivity: The derived sensitivity bound held under various calibration drift scenarios, with empirical bias staying well below the theoretical worst-case envelope.
• Real-World Case: In the UCI Adult experiment, the diagnostic correctly identified that the supervised baseline (OLS on $L$) outperformed pseudo-labelled estimators because the labelled set was sufficiently informative and the conditional probability was well-explained by the controls (low $V^*$ relative to the noise). Conversely, the diagnostic showed that hard labels on this dataset incurred minimal attenuation ($\hat{\kappa} \approx 0.92$) due to the richness of the classifier features.

6. Significance and Claims

The paper claims an operational contribution rather than a new algorithmic architecture. It does not propose a new pseudo-labelling algorithm or compete with state-of-the-art classification losses (like FixMatch). Instead, it provides:

1. A quantitative justification for why confidence thresholding biases downstream regression coefficients.
2. A diagnostic tool ($(V^*, \kappa)$ decision rule) that allows practitioners to compute, a priori, whether using hard labels is safe or if they should stick to soft scores or a purely supervised approach.
3. A structural insight that the separation between classifier features ($W$) and downstream controls ($X$) is not merely a design choice but a mathematical necessity for identification in this setting.

The authors emphasize that their framework is "calibration-aware," translating the abstract quality of a classifier's calibration into concrete bounds on downstream estimation bias, thereby bridging the gap between classification performance and causal/associational inference.