Selective prediction as a triage gate for primary-care depression screening: quantifying and mitigating selection bias in CHARLS-2011

Technical Summary: Selective Prediction as a Triage Gate for Primary-Care Depression Screening

Problem Statement
Primary care in China lacks structured mental health assessment tools, and existing machine learning (ML) models for depression screening are often developed on heavily selected samples. The authors argue that cumulative inclusion and exclusion criteria, typically treated as neutral data-cleaning steps, introduce a "dual distortion":

1. Epidemiological Distortion: Selection funnels compress or alter epidemiological associations (e.g., the relationship between chronic diseases and depression), potentially rendering findings from selected samples non-generalizable to the target population.
2. Metric Distortion: Machine learning performance metrics, particularly Area Under the Curve (AUC), are often inflated in selected samples due to reduced heterogeneity, while calibration error may degrade. This creates a misleading impression of model readiness for clinical deployment.

Furthermore, standard model-embedded uncertainty indicators often fail to effectively distinguish between "easy" and "difficult" cases in epidemiological data, limiting the utility of selective prediction (classification with a reject option).

Methodology
The study utilizes the CHARLS 2011 baseline dataset (n=17,705) with temporal external validation on the CHARLS 2018 wave. The methodology is structured around three objectives:

1. Quantifying Selection Bias (The Funnel):

   • A four-level selection funnel (L0–L3) was constructed to simulate standard epidemiological exclusion processes.
     • L0: Complete sample after Multiple Imputation by Chained Equations (MICE).
     • L1: Restricted to age ≥60.
     • L2: Required complete CES-D-10 responses.
     • L3: Required complete sociodemographic covariates (education, insurance, residence).
   • Association Analysis: Logistic regression models were fitted at each level to track the drift in Odds Ratios (OR) for five chronic diseases against depression.
   • Performance Analysis: Five classifiers (Logistic Regression, Random Forest, XGBoost, LightGBM, Gradient Boosting) were evaluated across levels using nested cross-validation and SMOTE. Monotonic trends in AUC, Brier score, and Expected Calibration Error (ECE) were tested using Page's L test.

2. Mitigating Bias via Selective Prediction:

   • Model-Embedded Uncertainty: The study tested whether each model's internal uncertainty score (e.g., predictive entropy, tree variance, quantile residual intervals) could effectively filter samples to improve AUC.
   • Decoupled Predictor–Selector Framework: To overcome the limitations of embedded scores, the authors proposed a decoupled approach where an independent selector assesses predictive reliability. Nine candidate selectors were evaluated, with XGBoost cross-validation residuals identified as the optimal external selector. This selector was applied to all five predictors to determine a "reliable" subset of the population.

3. Clinical Deployment via Interpretable Rules:

   • Risk Stratification: Samples in the final analytic set (L3) were stratified into low, medium, and high-risk groups based on robust residuals (median of five hyperparameter configurations).
   • CART Rule Extraction: A Classification and Regression Tree (CART) was fitted to the robust residuals using 15 clinical features to extract interpretable "white-box" rules defining the boundaries of the reliable prediction domain.
   • Stability Checks: Multi-parameter (5 configurations) and multi-seed (10 seeds) stability analyses were performed to ensure the risk stratification was a structural property of the data.

Key Results

• Selection Bias Distortion:

  • Sample size dropped from 17,705 (L0) to 4,256 (L3) (24.0%).
  • Association Attenuation: The OR for cancer-depression dropped from 1.78 (95% CI 1.32–2.41) at L0 to 1.39 (95% CI 0.74–2.63) at L3, losing statistical significance. Other associations showed varying degrees of compression.
  • Metric Inflation: AUCs for all five models increased directionally from L0 to L3 (e.g., XGBoost from 0.775 to 0.783), but these trends were not statistically significant after multiple-comparison correction. Conversely, calibration error (ECE) significantly increased for four of the five models as selection tightened.

• Selective Prediction Performance:

  • Embedded Uncertainty: Only XGBoost, using a custom out-of-fold quantile residual interval, achieved significant selective prediction gains (AUC ~0.83 at ~52% coverage). Other models' embedded indicators failed to identify an AUC peak better than full coverage.
  • Decoupled Framework: When using XGBoost cross-validation residuals as an external selector for all five models, all classifiers achieved consistent selective prediction gains.
    • Coverage: Approximately 20% of samples were identified as reliable.
    • Performance: Test AUCs ranged from 0.888 to 0.901 across all models at this coverage. Precision was ~0.85–0.86, and recall ~0.89–0.90.
    • Stability: The 20% coverage threshold was highly consistent across different model architectures, indicating the selector identifies a stable "reliable domain."

• Interpretable Triage Rules:

  • CART analysis identified four key variables for triage: Self-rated health, Education, Pain severity, and Marital status.
  • Low-Residual (Reliable) Domain: Characterized by higher education, specific pain profiles, or poor self-rated health with clear somatic drivers.
  • High-Residual (Uncertain) Domain: Characterized by lower education, complex psychosocial factors, and high depression prevalence (56.9%), where model predictions are unreliable.
  • External Validation: Applying the 2011-derived CART rules to the 2018 cohort showed stable group-level AUCs, confirming the temporal robustness of the triage logic.

Significance and Claims
The paper claims to provide a deployable primary-care triage pathway that addresses the "black-box" problem in ML-driven health screening. Its significance lies in three areas:

1. Methodological Correction: It demonstrates that cumulative selection bias produces a dual distortion (compressed associations and inflated metrics) that must be explicitly reported. The authors argue that performance metrics should always be reported with their selection level, coverage, and calibration trajectory.
2. Framework for Reliability: It establishes that decoupled selective prediction using cross-validation residuals is a robust method to identify a high-confidence subset of patients, outperforming model-embedded uncertainty indicators which often fail in epidemiological contexts.
3. Clinical Actionability: By converting complex ML uncertainty into a simple, four-variable CART rule, the study offers a practical "triage gate." This allows general practitioners to route approximately 20% of patients (the reliable domain) to algorithm-assisted scoring while directing the remaining 80% (the uncertain domain) to human evaluation, thereby ensuring patient safety and maintaining interpretability in resource-constrained settings.

The authors conclude that this approach transforms model uncertainty monitoring from a statistical post-processing step into a routine quality-management checkpoint for tiered healthcare systems.