Confronting Label Indeterminacy in Automated Bail Decisions

Technical Summary: Confronting Label Indeterminacy in Automated Bail Decisions

Problem Statement
Automated decision support systems for bail face a fundamental data limitation known as label indeterminacy. In pretrial settings, the counterfactual outcome—whether a defendant would have appeared in court if released—is unobserved when bail is denied and the defendant is detained. Consequently, historical datasets contain structural indeterminacy: the observed outcome (court appearance) is a direct consequence of the detention decision rather than the defendant's underlying propensity to appear. Training machine learning models on such data without addressing this indeterminacy risks embedding structural biases and reinforcing feedback loops, as the models learn from outcomes that are partially determined by the very decisions they aim to predict.

Methodology
The authors conducted a case study using 90,732 anonymized court cases from the Unified Judicial System of Pennsylvania (2016–2020). The target variable is Failure to Appear (FTA). The study defines "indeterminate" labels as cases where an intervention (pretrial detention) alters the observed outcome, rendering the defendant's true propensity unobservable. This includes cases where bail was denied, set but not paid, partially posted, or where bonds were terminated for reasons other than non-appearance.

To investigate the impact of handling these labels, the authors:

1. Defined Five Imputation Strategies:

   • Correct labels ($corr$): Retains original labels (appearance) for detained defendants, assuming they would have appeared if released.
   • Detention-as-failure ($daf$): Assigns an FTA label to all detained defendants, assuming detention reflects an accurate prediction of non-appearance.
   • Observed only ($obs$): Excludes all indeterminate cases, training only on cases where bail was granted.
   • Observed + Inverse Propensity ($obs\_ip$): Trains on observed cases but applies inverse propensity weighting to correct for sample bias, assuming all factors influencing bail decisions are captured in the data.
   • Nearest Neighbor ($nn$): Imputes labels for detained defendants based on the outcomes of the most similar cases where bail was granted ($k=52$).

2. Experimental Setup:

   • Three machine learning models were trained: Logistic Regression, Random Forest, and XGBoost.
   • The dataset was split into a test set (20%) and 25 balanced training subsets (via undersampling of the minority FTA class).
   • Sensitive attributes (race, sex) were retained to reflect real-world data distributions, while post-decision features (bail amount, status) were excluded.
   • A total of 348 models were trained (3 models × 5 methods × 25 subsets).

3. Evaluation:

   • Predictive performance was measured using the Matthew's Correlation Coefficient (MCC) on both determinate and indeterminate test cases.
   • Predictive behavior was analyzed via prediction distributions (violin plots) and Wasserstein distances.
   • Internal decision-making processes were examined using Explainable AI (XAI) techniques, specifically feature importance scores (gain metric) for the XGBoost model.

Key Results

• Impact of Imputation vs. Model Choice: The choice of label imputation method had a more substantial impact on model behavior and prediction distributions than the choice of the machine learning algorithm itself. Statistical tests (Wasserstein distance, Kolmogorov–Smirnov) confirmed that methods like $corr$, $daf$, and $nn$ produced meaningfully distinct distributions compared to $obs$ and $obs\_ip$.
• Performance Variance:
  • The $obs$, $obs\_ip$, and $nn$ methods performed better on determinate test cases but worse on indeterminate cases.
  • The $corr$ method performed worst on determinate cases but best on indeterminate cases, as it is the only method preserving the original labels for detained defendants.
• Internal Decision-Making: XAI analysis revealed that different imputation methods altered the internal feature importance rankings. While core features (e.g., leading offense, attorney type) remained significant across methods, the relative importance of other features shifted depending on the imputation strategy, indicating that label handling fundamentally changes how models utilize case information.
• Distribution Shifts: The $obs$ and $obs\_ip$ methods yielded similar prediction distributions, suggesting that inverse propensity weighting did not significantly alter the model's output in this specific context compared to simple exclusion.

Significance and Claims
The paper argues that label indeterminacy is not merely a technical data issue but a normative one. The authors claim that:

1. Technical Choices are Normative: Each imputation method encodes a specific conception of justice and risk governance. For instance, $corr$ implicitly validates preventive detention as a substitute for individualized risk assessment, while $daf$ treats past detention decisions as reliable evidence of risk, potentially creating self-fulfilling prophecies.
2. Unverifiable Assumptions: All methods rely on unverifiable assumptions about counterfactual outcomes. No single approach can be definitively proven correct, yet each significantly influences predictive behavior and fairness.
3. Accountability Risks: Statistical adjustments (like $obs\_ip$ and $nn$) may create a "technocratic confidence" where probabilistic inferences replace missing legal observations, potentially undermining the ability of defendants to contest decisions based on counterfactual labels.
4. Design Implications: The study demonstrates that methodological choices regarding label treatment can be more critical than model selection. Therefore, developers of bail decision support systems must explicitly acknowledge the legal and philosophical implications of how they handle indeterminate labels, rather than treating them as purely algorithmic problems.

The authors conclude that properly addressing label indeterminacy is critical for the legitimacy, fairness, and accountability of automated bail systems, urging a shift from viewing these as technical gaps to recognizing them as sites of legal and moral discourse.