Technical Summary: Hard and Soft Label Assignment for Cost-Sensitive Semi-Supervised Reject Inference in Credit Scoring

1. Problem Statement

Credit risk scorecards are traditionally trained exclusively on accepted applicants because only these individuals have recorded repayment outcomes. This practice introduces sample selection bias, causing models to perform well on the accepted population but poorly on the wider applicant pool. Furthermore, standard classification approaches often ignore the asymmetric costs of misclassification in credit scoring: the cost of a False Negative (approving a defaulter) is significantly higher than that of a False Positive (rejecting a good payer).

Existing reject inference methods often fail to address these issues simultaneously. Many assume balanced class distributions or treat misclassification errors as having equal cost, leading to suboptimal economic outcomes. Additionally, while semi-supervised learning treats rejected applicants as instances with missing labels, few frameworks provide formal theoretical guarantees regarding consistency under specific rejection mechanisms (e.g., Missing at Random) while integrating cost-sensitive objectives.

2. Methodology: RI-CSCEM

The paper proposes RI-CSCEM (Reject Inference - Cost-Sensitive Classification Expectation-Maximisation), a unified framework that integrates semi-supervised logistic regression with cost-sensitive learning. The method operates on a dataset $D = D_l \cup D_u$, where $D_l$ contains labelled (accepted) applicants and $D_u$ contains unlabelled (rejected) applicants.

2.1 Core Algorithm

The algorithm follows a Classification EM (CEM) procedure with two alternating steps:

• E-Step (Label Assignment):
  Instead of assigning a single hard label to rejected applicants, the method employs a soft responsibility mechanism. For each unlabelled applicant $j$, a cost-weighted difference criterion is calculated:
  $$\text{Diff}(j) = C_{FN} \log(1 + e^{-z_j}) - C_{FP} \log(1 + e^{z_j})$$
  where $z_j = \beta^\top x_j$, and $C_{FN}, C_{FP}$ are the costs of false negatives and false positives, respectively.

  This criterion is mapped to a soft default probability $q_j \in (0, 1)$ using a sigmoid function calibrated by a temperature parameter $\tau$. Crucially, the assignment is constrained by a balance constraint: the expected proportion of defaults among the rejected set must equal the empirical default rate $r$ observed in the accepted set ($D_l$). This prevents degenerate solutions where all rejects are assigned to a single class.

  A hard-label variant is also defined as a special case, where the $\lfloor r \cdot m \rfloor$ rejected applicants with the lowest cost-weighted loss are assigned a hard label of 1 (default), and the rest 0.

• M-Step (Parameter Estimation):
  The model refits a cost-sensitive $\ell_1$-penalised logistic regression on the augmented dataset. The objective function $L(\beta, q)$ combines:

  1. Weighted logistic loss on labelled data ($D_l$).
  2. Soft-assigned weighted loss on unlabelled data ($D_u$), scaled by a hyperparameter $\lambda \in [0, 1]$.
  3. An $\ell_1$ penalty ($\mu \|\beta\|_1$) for feature selection and sparsity.

  The unlabelled data contribution is scaled by $\lambda$, allowing interpolation between purely supervised ($\lambda=0$) and fully semi-supervised ($\lambda=1$) regimes.

2.2 Theoretical Guarantees

The paper provides two key theoretical results:

1. Monotone Descent: For the hard-label variant, the alternating E- and M-steps guarantee a monotonically non-increasing objective function, ensuring convergence.
2. Consistency under MAR: Under the Missing at Random (MAR) assumption—where the rejection decision depends only on observed covariates and not the unobserved default outcome—the RI-CSCEM estimator is consistent. The authors prove that under MAR, the accepted-only estimator is already unbiased; thus, the reject inference procedure does not asymptotically improve the estimator but serves as a diagnostic tool to measure when reject inference adds value (specifically when MAR does not hold).

3. Key Contributions

1. Unified Framework: RI-CSCEM unifies semi-supervised learning and cost-sensitive learning within a single penalised likelihood framework, addressing both sample selection bias and class imbalance simultaneously.
2. Soft vs. Hard Assignment: The paper introduces a soft responsibility assignment that distributes rejected applicants across classes based on cost-weighted criteria. Empirical results show this preserves discrimination power that hard pseudo-labelling destroys.
3. Theoretical Rigor: The work establishes formal guarantees of monotone descent for the hard variant and consistency under MAR, providing a controlled setting to evaluate the necessity of reject inference.
4. Rigorous Benchmarking: The study employs a robust experimental design involving 30 stratified draws and statistical testing (Friedman and Nemenyi tests) against 12 competing models, including traditional reject inference, resampling strategies, and semi-supervised baselines.

4. Experimental Results

The method was evaluated on Lending Club data (approx. 1.27 million accepted and 2.06 million rejected applications).

• Performance: Across 30 stratified draws, RI-CSCEM achieved a mean total misclassification cost of 1120.3.
• Comparison: This performance was statistically indistinguishable from the best accepted-only baselines: Cost-Sensitive Logistic Regression (1070.4) and SMOTE-balanced Logistic Regression (1104.7).
• Ablation Study:
  • Cost-Sensitive Weighting: Moving from a base CEM model to cost-sensitive weighting (CEM+CS) significantly reduced cost (from 1401.5 to 1215.6) and moved the model away from the trivial "all-non-default" solution.
  • Soft Responsibility: Replacing hard labels with soft responsibilities (the full RI-CSCEM) further reduced cost to 1120.3 and recovered discrimination, raising the AUC from 0.589 to 0.614.
  • Hard vs. Soft: The hard-label variant (RI-CSCEM-hard) performed similarly to CEM+CS (1214.3 vs 1215.6), confirming that the soft assignment is the primary driver of the AUC improvement.
• Statistical Significance: The Friedman test rejected the null hypothesis of equal performance across all models ($p < 10^{-3}$). The Nemenyi post-hoc test confirmed that RI-CSCEM belongs to the same statistically significant group as the top-performing accepted-only baselines.

5. Significance and Claims

The paper claims that RI-CSCEM provides a diagnostic baseline for the credit scoring literature. Its primary significance lies in demonstrating that under a Missing at Random (MAR) rejection mechanism, reject inference adds little value over a well-tuned, cost-sensitive accepted-only model. The authors argue that the accepted-only estimator is already unbiased under MAR, and the proposed framework confirms this theoretically and empirically.

The paper modestly concludes that the true benefit of reject inference should emerge in Missing Not at Random (MNAR) scenarios, where rejection depends on unobserved factors. The current work establishes a rigorous foundation for such future extensions by proving consistency under MAR and isolating the specific components (cost-sensitivity and soft assignment) that improve model performance when reject inference is applied. The authors do not claim that RI-CSCEM universally outperforms all baselines in all regimes, but rather that it matches the strongest baselines under MAR while offering a robust, theoretically grounded mechanism for handling unlabelled data when the rejection mechanism is ignorable.