Auxiliary-Enhanced Survey Calibration: A Novel Framework for Reject Inference under Sample Selection Bias

Technical Summary: Auxiliary-Enhanced Survey Calibration for Reject Inference

Problem Statement
Credit scoring models are typically trained on accepted applicants, creating a sample selection bias where the training distribution $P(Y|X, Z=1)$ differs from the target population distribution $P(Y|X)$. This phenomenon, known as the reject inference problem, arises because lenders systematically reject applicants they deem high-risk, leading to a training set enriched with "good payers." Consequently, models trained solely on accepted data tend to underestimate default rates in the full applicant pool. Existing reject inference methods generally fall into two categories: those that impute missing outcomes for rejected applicants (e.g., augmentation, reclassification, generative models) and those that reweight the accepted sample (e.g., Inverse Propensity Score weighting). The authors argue that imputation-based methods are fundamentally flawed because they fabricate labels using a model already corrupted by selection bias, thereby propagating and amplifying the distortion. Conversely, standard reweighting methods often suffer from instability due to extreme weights when the positivity assumption is violated.

Methodology
The paper proposes a novel framework based on survey calibration theory (Deville and Sarndal, 1993) that avoids outcome imputation entirely. The core mechanism involves computing observation-level weights for accepted applicants to force the weighted sample's covariate distribution to match the known population totals of auxiliary variables.

1. Framework Setup:

   • Let $U$ be the full population of applicants, $A$ the accepted set, and $R$ the rejected set.
   • Features are partitioned into common variables $X^{(c)}$ (observed for all applicants, used for weighting) and uncommon/underwriting variables $X^{(u)}$ (observed only for accepted applicants, used for the final outcome model).
   • The goal is to estimate the population parameter $\theta^*$ by maximizing a weighted log-likelihood on the accepted set $A$.

2. Calibration Weights:

   • Weights $w_i$ are derived by solving a constrained optimization problem that minimizes a distance function $D(w_i, d_i)$ (where $d_i$ is a design weight, typically 1) subject to the constraint that the weighted sum of common variables equals the known population totals: $\sum_{i \in A} w_i X^{(c)}_i = T^{(c)}$.
   • The authors utilize bounded distance functions (specifically the bounded logit and truncated-linear distances) rather than unbounded ones (like raking/exponential). This ensures weights remain within a user-specified interval $[L, U]$, preventing the effective sample size from collapsing and avoiding the instability associated with Inverse Propensity Score (IPS) weighting.

3. Estimation:

   • The weights are computed using a Newton-Raphson algorithm to solve for Lagrange multipliers.
   • The final logistic regression model is fitted by maximizing the calibration-weighted log-likelihood: $\tilde{\ell}(\theta; w) = \sum_{i \in A} w_i [Y_i(\beta_0 + \beta^\top X_i) - \log(1 + \exp(\beta_0 + \beta^\top X_i))]$.
   • Crucially, the weighting step uses only $X^{(c)}$, while the outcome model utilizes the full feature vector $X = (X^{(c)}, X^{(u)})$. This allows the method to leverage richer underwriting data available only for accepted applicants, a capability imputation methods lack.

Key Contributions

1. Imputation-Free Framework: The authors introduce a reject inference estimator that fabricates no labels. By relying solely on reweighting, it avoids inheriting the bias of the initial accepted-only model, a flaw inherent in augmentation and reclassification methods.
2. Theoretical Guarantees: The paper proves that the calibration-weighted estimator is consistent and $\sqrt{n}$-asymptotically normal under the Missing At Random (MAR) assumption. It derives a closed-form asymptotic variance and a sandwich variance estimator to account for the uncertainty introduced by weight estimation.
3. Population Projection Characterization: The authors clarify that under model misspecification, the accepted-only estimator converges to the selected-sample projection of the truth, whereas the calibration estimator targets the population projection. This distinction is where the method provides value over the baseline.
4. Weight Stability: By employing bounded distance functions, the method avoids the extreme weights and effective sample size collapse that plague unbounded reweighting techniques like IPS and raking.

Experimental Results
The study evaluates the method against nine strategies (including fuzzy augmentation, reclassification, IPS, Heckman two-step, and generative models) across three datasets: the Lending Club dataset, a semi-synthetic Taiwan Credit benchmark (with ground-truth labels for rejected applicants), and controlled simulations.

• Discrimination (AUC): On the Lending Club dataset, the calibration method held discrimination at the baseline level (AUC $\approx$ 0.703 vs. 0.706 for the baseline using full features). In contrast, imputation-based methods degraded performance, losing up to 11 AUC points in the Taiwan benchmark and 3.6 points on Lending Club. The authors attribute the slight gains in full-feature models to the ability to use underwriting variables ($X^{(u)}$), which imputation methods cannot access for rejected applicants.
• Calibration (Risk Estimation): The primary gain of calibration is in population risk estimation. On Lending Club, calibration reduced the Expected Calibration Error (ECE) by nearly three-fold (from 0.0112 to 0.0042) compared to the baseline. In the Taiwan misspecification regime, the reduction was six-fold. Imputation methods significantly increased calibration error (up to 16-fold).
• Misspecification: In simulation studies where the model was misspecified (omitting an interaction term), the calibration estimator remained close to the oracle (population truth), while the accepted-only baseline diverged significantly as selection strength increased.
• MNAR Limitation: As expected, the method failed to recover population parameters under Missing Not At Random (MNAR) selection (where rejection depends on the unobserved outcome), confirming the theoretical boundary of the approach.

Significance and Claims
The paper claims that calibration offers a safe default for reject inference. Because setting all weights to one recovers the baseline exactly, the method cannot perform worse than the accepted-only model. Its significance lies in:

• Correcting Population Risk: It provides a statistically valid way to estimate population-level default rates and calibration metrics, which are critical for pricing and capital provisioning, without the risk of bias propagation inherent in imputation.
• Robustness to Misspecification: It outperforms the baseline when the working model is misspecified, effectively estimating the population parameter rather than the biased selected-sample parameter.
• Practical Stability: The use of bounded distances resolves the numerical instability common in other reweighting approaches.

The authors conclude that while the method does not solve MNAR selection or eliminate the need for common support between accepted and rejected applicants, it represents a superior alternative to imputation-based strategies, particularly for ensuring the reliability of population-level risk estimates in credit scoring.