Probabilistic Risk Stratification and Transition-Sensitive Explainability for Attorney Involvement in Workers' Compensation Claims

Technical Summary: Probabilistic Risk Stratification and Transition-Sensitive Explainability for Attorney Involvement in Workers' Compensation Claims

Problem Statement
Attorney involvement in workers' compensation claims is a critical operational challenge, consistently associated with increased resolution times, higher costs, and administrative complexity. Prior research has predominantly treated legal representation as a binary outcome (involvement vs. no involvement) and relied on retrospective statistical analyses that assume a uniform escalation mechanism across all claims. This binary framing fails to capture the heterogeneous nature of claim evolution, where risks vary significantly across the population, and offers limited guidance for operational prioritization. Furthermore, existing machine learning applications in this domain often prioritize classification accuracy over the reliability of probability estimates, neglecting the need for well-calibrated risk scores that can support continuous ranking and early intervention strategies.

Methodology
The study proposes an integrated analytical framework that models attorney involvement as a probabilistic risk process rather than a fixed classification task. The methodology proceeds through the following stages:

1. Data and Cohort Construction: The analysis utilizes the Structured and Textual Records for Injury Data Exploration (STRIDE) database, focusing on a filtered cohort of 48,130 lost-time claims (excluding medical-only claims) from 32 states. To ensure prospective validity and prevent target leakage, the model utilizes only features available early in the claim lifecycle (e.g., injury descriptors, demographics, jurisdiction, initial wage data), explicitly excluding post-escalation variables such as legal identifiers, final costs, and closure dates.
2. Probabilistic Modeling: A CatBoost classifier is employed as the primary modeling framework due to its native handling of categorical variables and ordered boosting capabilities. The model is trained to output continuous probability estimates of attorney involvement rather than binary class labels. Hyperparameter tuning is conducted via five-fold stratified cross-validation on the training set, optimizing for ROC-AUC.
3. Model Evaluation: Performance is assessed on a held-out test set using discrimination metrics (ROC-AUC, PR-AUC) and calibration metrics (Brier score, Expected Calibration Error). This ensures that the predicted probabilities are not only rank-consistent but also statistically reliable estimates of risk frequency.
4. Risk Stratification: Instead of applying a fixed decision threshold, claims are partitioned into three risk regimes based on rank position: Low-risk (bottom 20%), Medium-risk (middle 60%), and High-risk (top 20%). These regimes serve as analytical strata to investigate how risk mechanisms differ across the spectrum.
5. Explainability Framework:
   • Global Explainability: SHAP (Shapley Additive Explanations) values are computed to identify global feature importance. Robustness is verified through cross-model comparisons with XGBoost, Random Forest, Extra Trees, and Decision Trees.
   • Regime-Conditioned Explainability: SHAP analysis is conducted separately within each risk regime to uncover how feature contributions (magnitude and direction) vary across low, medium, and high-risk groups.
   • Transition-Oriented Analysis: A boundary sensitivity framework distinguishes between "frontier" claims (those near regime boundaries) and "stable" claims (those deep within a regime). By comparing SHAP values between these groups, the study identifies features driving transitions between risk levels (e.g., Low $\to$ Medium, Medium $\to$ High, and de-escalation Medium $\to$ Low).

Key Results

• Model Performance: The CatBoost model demonstrates strong discrimination (Test ROC-AUC: 0.7540; PR-AUC: 0.7148) and excellent calibration (Brier score: 0.1988; ECE: 0.0114), confirming that the probability outputs are reliable for risk ranking.
• Global Drivers: The most influential predictors identified globally include age_at_hire, injury_severity, age_at_injury, bdyprt_desc (body part description), and juris_st (state jurisdiction). Cross-model analysis confirms these rankings are stable across different tree-based algorithms.
• Heterogeneous Feature Effects: The regime-conditioned analysis reveals that feature effects are not uniform. For instance, injury_severity acts as a negative contributor (risk-reducing) in the low-risk regime, becomes neutral in the medium-risk regime, and becomes a strong positive driver (risk-increasing) in the high-risk regime.
• Transition Dynamics: The boundary analysis uncovers asymmetric mechanisms for risk escalation and de-escalation:
  • Low $\to$ Medium Transition: Driven primarily by juris_st (jurisdiction) and injury_severity, where frontier claims exhibit weaker risk-reducing contributions compared to stable low-risk claims.
  • Medium $\to$ High Transition: Dominated by injury_severity and juris_st, where frontier claims show a distinct shift toward positive contributions from these clinical and institutional variables.
  • Medium $\to$ Low (De-escalation): Characterized by a broad reduction in risk contributions across multiple features (including body part and tenure variables), often involving sign reversals, rather than a single dominant driver.
• Error Distribution: Misclassification is not uniform; the medium-risk regime contains the highest concentration of both false positives and false negatives, indicating it functions as a transition corridor with higher structural uncertainty.

Significance and Contributions
The paper claims five primary contributions to the field of workers' compensation analytics and machine learning:

1. Reframing the Problem: It shifts the perspective of attorney involvement from a binary classification problem to a probabilistic ranking problem aligned with operational decision-making needs.
2. Probabilistic Rigor: It incorporates model calibration and probabilistic evaluation (Brier score, ECE) to ensure that risk scores are reliable estimates of escalation likelihood, not just rankers.
3. Robust Feature Validation: It verifies the stability of key predictors across multiple learning algorithms, confirming that the identified drivers are structural rather than model-specific artifacts.
4. Risk-Stratified Explainability: It introduces a framework for analyzing SHAP values within specific risk regimes, revealing that the drivers of escalation change depending on the stage of risk (e.g., injury severity matters differently in low vs. high-risk claims).
5. Boundary-Focused Analysis: It provides a novel transition-oriented analysis of claims near regime boundaries, demonstrating that escalation and de-escalation follow asymmetric mechanisms. This offers a more nuanced understanding of how claims move through the risk spectrum compared to static global summaries.

The study concludes that modeling litigation as a probabilistic risk process with identifiable transition zones provides a structured view of attorney involvement, offering actionable insights for targeted intervention and resource allocation in workers' compensation systems.