Whose Truth Is Ground Truth?: Consequences of Label Choice on Machine Learning Models Predicting Depression 

Technical Summary: Whose Truth Is Ground Truth? Consequences of Label Choice on Machine Learning Models Predicting Depression

Problem Statement
Machine learning (ML) models developed using electronic health record (EHR) data frequently rely on provider-documented diagnoses (e.g., ICD codes) as the "ground truth" for training. However, mental health diagnosis, particularly for depression, is inherently subjective and relies heavily on patient self-report and provider-patient communication. There is substantial variation in how these conditions are diagnosed and recorded, and significant disagreement often exists between patient-reported symptoms (e.g., via PHQ-9) and provider-coded diagnoses. The paper posits that relying exclusively on provider labels may introduce bias, obscure patient experiences, and lead to models that reflect clinical workflow disparities rather than true patient states. The study addresses the critical question: How does the choice of depression label source (patient-reported, provider-coded, or concordant) impact the performance and feature importance of tree-based ML models?

Methodology

• Data Source: The study utilized a retrospective cross-sectional dataset from the OCHIN ADVANCE network, comprising 644,387 adults (aged 18+) with ambulatory or telehealth visits between 2012 and 2019. The data included demographics, vital signs, chronic conditions, social determinants of health (SDOH), and patient-reported outcomes.
• Label Strategies: Three distinct definitions of depression were constructed to serve as the target variable (ground truth):
  1. Provider-Indicated: Depression defined solely by ICD codes in the EHR.
  2. Patient-Indicated: Depression defined by "Most Severe" self-reported PHQ-9 scores (26 or 27), representing high certainty of symptoms without a corresponding provider code.
  3. Concordant: Depression defined by the intersection of "Most Severe" PHQ-9 scores and provider ICD codes.
     Note: Additional models (Tree 2b and 3b) were tested using a traditional PHQ-9 cutoff of 20 ("Severe") to address class imbalance issues.
• Predictors: Features included demographics, vital signs, chronic conditions (e.g., diabetes, hypertension), psychiatric diagnoses, and SDOH.
• Modeling Approach: Classification and Regression Trees (CART) were employed due to their interpretability. Models were trained using 70/30 splits with 10-fold cross-validation. Hyperparameters were tuned to maximize the Area Under the Curve (AUC).
• Evaluation Metrics: Performance was assessed using AUC, sensitivity, specificity, Positive Predictive Value (PPV), and F1 score.
• Explainability: SHapley Additive exPlanations (SHAP) were used to quantify and visualize feature importance across the different labeling strategies.

Key Results

• Model Performance: Overall model performance was modest, with AUC values ranging from 0.62 to 0.64 across all labeling strategies.
  • Provider-Indicated Model (Tree 1): Achieved the highest sensitivity (0.33) in the holdout test set among the primary models, with an AUC of 0.63.
  • Patient-Indicated Model (Tree 2): Showed lower sensitivity (0.28) and very low PPV (0.03) in the holdout set, likely due to the extreme rarity of the "Most Severe" class (1.3% of the sample).
  • Concordant Model (Tree 3): Demonstrated higher sensitivity in cross-validation (0.40) but lower performance in the holdout set (0.30 sensitivity, 0.01 PPV).
  • Alternative Cutoffs: When the "Most Severe" cutoff was relaxed to the standard "Severe" cutoff (PHQ-9 ≥ 20), sensitivity for patient-based models improved (Tree 2b: 0.49; Tree 3b: 0.37), though they still did not surpass the provider model in specificity or F1 scores.
• Feature Importance Disparities: SHAP analysis revealed that feature importance shifted substantially depending on the label source:
  • Anxiety: The only feature consistently influential across all three models.
  • Demographics: In the Provider-Indicated model, demographic features (specifically White race and female gender) were highly influential. In contrast, these demographic features were less important in Patient-Indicated and Concordant models.
  • Other Features: For patient-indicated models, younger age and blood pressure were more prominent. For the concordant model, the absence of hypertension and heterosexual orientation were top features.
• Bias Implications: The reliance on provider labels resulted in models that heavily weighted demographic variables, raising concerns that such models may amplify existing provider biases rather than capturing the underlying pathology of depression.

Key Contributions

1. Empirical Demonstration of Label Sensitivity: The study provides empirical evidence that the choice of ground truth label (patient vs. provider vs. concordant) fundamentally alters both model performance metrics and the interpretation of feature importance.
2. Highlighting Ground Truth Instability: It challenges the assumption that provider-coded diagnoses are a neutral or superior ground truth for mental health, demonstrating that ignoring patient-reported outcomes can lead to models that reflect clinical documentation biases rather than patient reality.
3. Demographic Bias in Labeling: The findings suggest that models trained on provider labels may inadvertently learn to predict the likelihood of a provider making a diagnosis (which correlates with race and gender) rather than the presence of the condition itself, as evidenced by the high influence of demographic features in the provider-labeled model.
4. Methodological Framework: The paper proposes a systematic approach for evaluating ML models in mental health by comparing multiple labeling strategies and their resulting explainability profiles to ensure equitable and trustworthy AI.

Significance and Claims
The authors claim that their findings underscore the necessity of transparent reporting and the inclusion of patient-reported outcomes (PROs) in model development to support equitable mental health AI. They argue that "trustworthy" models in mental healthcare must incorporate the patient's perspective, as diagnosis relies on symptom reporting rather than objective biomarkers.

The paper concludes that while provider-derived labels may yield slightly higher AUC or sensitivity in some contexts, this gain is offset by the amplification of demographic feature importance, which raises significant concerns about bias and generalizability. The study advocates for a shift away from relying solely on provider-coded diagnoses as ground truth. Instead, it calls for researchers to routinely compare labeling strategies to surface biases in seemingly well-performing models and to align ML development with measurement-based care guidelines that prioritize the patient's voice. The authors do not claim to have optimized a specific algorithm for clinical deployment but rather to have exposed the foundational risks of current labeling practices in mental health ML.