🤖 machine learning

Auditing Construct Overlap in Explainable Machine Learning: Evidence from Burnout-Depression Prediction Across Student Cohorts

This paper demonstrates that seemingly robust and stable risk hierarchies in explainable machine learning models for predicting burnout and depression are largely artifacts of construct overlap between correlated predictors and outcomes, a finding revealed through residualization experiments that show predictive performance collapses when shared variance is removed.

Original authors: Alireza Dehghan, Negin Ashrafi

Published 2026-07-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Alireza Dehghan, Negin Ashrafi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've built a super-smart robot detective to solve a mystery: "Who among the students is most likely to feel burned out and depressed?" You feed the robot a mountain of survey data, and it comes back with a very confident answer. It points to a finger and says, "It's Trait Anxiety! That's the number one clue! And right behind it is Health Satisfaction!"

The robot seems to have found a golden rule. You test this rule on different groups of students—freshmen, seniors, medical students, and even students from totally different majors. Every single time, the robot gives the exact same answer: Anxiety is #1, Health is #2. It looks like a rock-solid discovery, a universal law of student stress.

But here's the twist the paper reveals: The robot isn't actually a genius detective. It's a bit of a trickster, and it's falling for a very clever optical illusion.

The "Double-Dipping" Trap

The paper argues that the robot's "great discovery" is actually a construct overlap. Think of it like this:

Imagine you are trying to predict how "soggy" a sandwich is.

Clue A: You measure how much water is in the bread.
Clue B: You measure how much water is in the lettuce.
The Outcome: You define "Sogginess" as the total water in the bread plus the water in the lettuce.

If you ask a robot to predict "Sogginess" using "Water in Bread" as a clue, the robot will scream, "Water in Bread is the #1 most important clue!" And it will be right. But is it a deep, magical insight into the nature of sandwiches? No. It's just a tautology. The clue is part of the answer. The robot is just noticing that the clue and the answer are made of the same stuff.

In this study, the "Sogginess" is a score called the Burnout-Depression Composite Index (BDCI). This score is built by adding up several parts, including a specific test for Depression (called CES-D).
The robot's #1 clue is Trait Anxiety (STAI-T).

Here is the problem: In the real world, anxiety and depression are best friends. They hang out together so much that they are highly correlated (a score of 0.72). Because the "Depression" part is baked inside the final "Burnout-Depression" score, and because Anxiety is so closely linked to Depression, the robot just sees Anxiety and thinks, "Aha! That's the cause!"

The paper shows that the robot isn't learning a complex, portable rule about student life. It's just noticing that Anxiety and Depression are twins, and since the final score includes one twin, the other twin looks like the most important predictor.

The "Magic Eraser" Experiment

To prove this, the authors didn't just guess; they ran a "residualization" audit. Think of this as a Magic Eraser that wipes away the specific connection between Anxiety and Depression.

The First Erase: They took the "Anxiety" clue and scrubbed away everything that looked like "Depression." They asked the robot to predict the score using only the unique parts of Anxiety that aren't depression.
- The Result: The robot's confidence crashed. Its accuracy (R²) dropped from a strong 0.41 down to a weak 0.16. The "Anxiety" clue fell from being the #1 star to being the #6th best clue.
The Second Erase: They took the final "Burnout-Depression" score and scrubbed away the "Depression" part entirely, leaving only the pure "Burnout" parts.
- The Result: The robot completely lost its mind. Its accuracy plummeted to 0.016. It was basically guessing.

This proves that the "stable" hierarchy the robot found wasn't a deep truth about student stress. It was an artefact. The robot was just riding the wave of the fact that Anxiety and Depression are so tightly linked. Once you cut that link, the "magic" disappears.

The "Fuzzy Crystal Ball"

Even if the robot could predict perfectly, the paper drops another bombshell: It's too fuzzy to be useful for individuals.

The authors used a special technique called "conformal prediction" to draw a safety net around the robot's guesses. They found that for any single student, the robot's prediction comes with a huge error bar.

The average width of this safety net is 35.4 units on a 0–100 scale.
To put that in perspective, the entire range of possible scores is 100. The robot's "guess" is so wide that it's 2.4 standard deviations off.

Imagine a weather app that says, "Tomorrow's temperature will be between 10°F and 80°F." That's technically a prediction, but it's not actionable. You can't decide what to wear based on that. The paper concludes that with the current data, individual-level clinical prediction is not actionable. The robot is too blurry to tell you if your specific friend is at risk.

The One Real Hero

So, if Anxiety is a trick and the robot is too fuzzy, is there anything useful?
Yes! When the authors wiped away the Anxiety-Depression link, one clue stood tall and true: Health Satisfaction.
When the "depression noise" was removed, students who were happy with their health were the best predictors of the remaining burnout signals. This suggests that how you feel about your own health might be a more genuine, portable clue than your anxiety levels.

The Bottom Line

The paper isn't saying the robot is "bad" or that anxiety doesn't matter. It's saying that apparent stability isn't always a discovery. Just because a pattern repeats across different groups doesn't mean you've found a new law of physics; sometimes, you've just found a mirror reflecting the same correlation over and over.

The authors suggest that before we trust any "AI detective" that claims to find the #1 cause of a problem, we must run this "Magic Eraser" test first. If the clue disappears when you remove the overlap with the answer, the clue was never really a clue at all—it was just a shadow.

What the paper rules out:

It rules out the idea that the "Anxiety is #1" finding is a portable, generalizable risk structure for burnout.
It rules out the possibility of using this specific setup to make individual clinical predictions (the error bars are too wide).
It rules out the idea that the model learned a complex, deep relationship; it learned a simple, linear correlation between two overlapping concepts.

What the paper suggests:

It suggests that Health Satisfaction might be a more robust, independent predictor of burnout once the depression link is broken.
It suggests that future studies must use this "residualization" check to avoid fooling themselves with instrument correlations.

The paper doesn't claim to have solved student burnout. It claims to have found a glitch in the matrix of how we use AI to study it, and it offers a simple tool to fix the glitch before we build our next generation of mental health tools.

Technical Summary: Auditing Construct Overlap in Explainable Machine Learning

Problem Statement
Explainable machine learning (XML) pipelines applied to composite mental health outcomes often generate apparently robust, cross-population-stable risk hierarchies. A critical concern is whether these stable rankings reflect genuine, portable risk structures or are merely artifacts of how the outcome variable was constructed. Specifically, when a predictor variable and a component of the outcome variable share a latent construct (construct overlap), the predictor's apparent explanatory dominance may be a psychometric artifact rather than a true causal or predictive relationship. This study investigates this phenomenon in the context of predicting burnout-depression severity among medical and non-medical students, where trait anxiety (STAI-T) and the depression scale (CES-D) are highly correlated ( $r=0.72$ ), and CES-D is embedded within the composite outcome.

Methodology
The study utilizes three datasets sharing identical instruments: a primary cross-sectional cohort of 886 medical students (University of Lausanne, 2022), a longitudinal cohort of 2,580 medical student observations across three time points, and a cross-sectional cohort of 701 non-medical students from eight faculties.

Outcome Construction: The authors constructed a Burnout–Depression Composite Index (BDCI) on a 0–100 scale, integrating three Maslach Burnout Inventory–Student Survey (MBI-SS) subscales and the CES-D. The index applies a severity-weighting logic where components exceeding the 75th percentile are doubled.
Prediction Pipeline: An ElasticNet regression pipeline was trained on the primary cohort (80/20 split) using 12 predictors (demographics, health satisfaction, psychotherapy history, and STAI-T). SHAP (SHapley Additive exPlanations) values were computed to establish feature importance hierarchies.
Cross-Cohort Validation: A reduced 7-feature model was trained on the 2022 data and applied without retraining to the temporal and non-medical cohorts to test for stability.
Residualization Audit (Core Contribution): To isolate the effect of construct overlap, two orthogonality experiments were conducted:
- Predictor Residualization: STAI-T was replaced by the residuals of regressing STAI-T on CES-D, isolating anxiety variance orthogonal to depression.
- Outcome Residualization: The MBI-SS burnout subscales were individually residualized against CES-D to create a "depression-free" composite (BDCI-C).
Uncertainty Quantification: Split conformal prediction was used to generate 90% prediction intervals to assess the feasibility of individual-level clinical deployment.

Key Results

Apparent Stability: The standard pipeline produced a highly stable feature hierarchy across all five evaluation sets (primary test, three longitudinal waves, and non-medical cohort). Trait anxiety (STAI-T) consistently ranked first, followed by health satisfaction, with a Kendall $\tau = 1.0$ for the top two positions. The model achieved $R^2$ values between 0.41 and 0.49 across cohorts.
The Collapse: The residualization experiments revealed that this stability was an artifact of the STAI-T/CES-D correlation:
- When STAI-T was residualized against CES-D (removing the shared variance), $R^2$ dropped from 0.414 to 0.156, and STAI-T fell from rank 1 to rank 6.
- When the burnout outcome (MBI-SS) was residualized against CES-D (BDCI-C), $R^2$ collapsed to 0.016. The model could essentially not predict the burnout-specific variance using the available feature set.
Uncertainty: Conformal prediction intervals averaged 35.4 units on the 0–100 scale (2.4 outcome standard deviations). This width independently rules out the feasibility of individual-level clinical prediction with the current feature sets, regardless of the construct overlap issue.

Key Contributions

Residualization Protocol: The paper introduces a transferable audit protocol for XAI studies. By regressing correlated predictors against outcome components (or vice versa) and re-evaluating model performance, researchers can quantify how much of an apparent "stable hierarchy" is driven by psychometric overlap rather than genuine risk relationships.
Re-evaluation of Stability: The study demonstrates that cross-cohort stability in XAI pipelines is not sufficient evidence of a portable risk structure when the signal is anchored to a fixed, high-correlation psychometric relationship (e.g., $r=0.72$ between anxiety and depression).
Identification of Robust Predictors: After removing the anxiety-depression channel, health satisfaction emerged as the most robust predictor (rank 1), suggesting it may represent a more independent burnout pathway than trait anxiety.

Significance and Claims
The authors claim that standard XAI pipelines cannot distinguish between genuine clinical signals and psychometric artifacts without explicit orthogonality checks. The "robust" risk hierarchies often reported in literature may simply reflect instrument correlation. The paper argues that any XAI study combining correlated predictor and outcome constructs must apply this residualization check before interpreting feature stability as a scientific finding.

Furthermore, the study establishes a separate, independent barrier to deployment: even if construct overlap were resolved, the current uncertainty (prediction intervals of ~35 units) renders individual-level clinical prediction non-actionable. The authors conclude that researchers who skip the residualization audit risk describing instrument correlation rather than replicated science. The paper does not claim to have solved the prediction problem but rather to have provided a necessary diagnostic tool to prevent misinterpretation of existing models.