Building a prediction model for outcomes following treatment in UK NHS Talking Therapies services for depression and anxiety

This study developed and internally validated multivariable prediction models using routine data from nearly 31,000 UK NHS Talking Therapies patients to identify key predictors of treatment outcomes for depression and anxiety, demonstrating the potential for data-driven tools to support personalized care and early intervention for those at risk of poor recovery.

The Gist

Imagine you walk into a massive, busy library (the NHS Talking Therapies service) looking for a book that will help you solve a specific problem, like feeling down (depression) or worried (anxiety). The librarians (therapists) are experts, and they have a proven system to help people. However, even with the best system, about half the people leave the library feeling like the book didn't quite fix their problem.

This paper is like a team of data detectives trying to build a "Crystal Ball" for these librarians. Their goal? To look at a person's profile before they start their therapy and predict, with reasonable accuracy, whether that specific therapy will work for them or if they might need a different kind of help.

Here is the breakdown of their work, using simple analogies:

1. The Mission: Why Build a Crystal Ball?

Currently, the library treats everyone roughly the same way. If you have anxiety, you get the standard anxiety course. But just like two people with the same cold might need different medicines, two people with the same anxiety might need different therapy approaches.

The researchers wanted to stop guessing and start knowing. They asked: "Can we look at a person's background and symptoms today to tell us if they are likely to get better with the standard plan, or if they are at risk of struggling?"

2. The Ingredients: What Data Did They Use?

They didn't use magic; they used a massive amount of real-world data. They looked at 31,000 adults who went through high-intensity therapy in South London between 2018 and 2024.

Think of this data as a giant recipe book. They looked at:

• The Symptoms: How bad was the depression or anxiety at the start? (Like measuring the temperature of a fever).
• The Life Context: Was the person unemployed? Did they have a long-term illness? Were they receiving government benefits? Could they speak English fluently?
• The History: Had they tried therapy before? Were they taking medication?

3. The Cooking Method: How Did They Build the Model?

The researchers used a sophisticated computer algorithm called Elastic Net.

• The Analogy: Imagine you are trying to bake the perfect cake. You have 50 ingredients (predictors). Some are crucial (flour, eggs), and some don't matter much (a pinch of cinnamon).
• The computer tried thousands of combinations to figure out which ingredients actually made the cake rise (recovery) and which ones made it fall flat (poor outcome).
• They didn't just guess; they used a technique called Bootstrap Resampling.
  • The Analogy: Imagine tasting your soup 200 times, but each time you take a slightly different spoonful from the pot. If the soup tastes good every single time, you know your recipe is solid. If it tastes terrible half the time, your recipe is shaky. This ensured their "Crystal Ball" wasn't just lucky; it was reliable.

4. The Results: How Good is the Crystal Ball?

The results were promising, though not perfect.

• The Score: They measured success using a score called AUC (Area Under the Curve). A score of 0.5 is a coin flip (50/50 chance). A score of 1.0 is a perfect prediction.
• The Outcome: Their models scored between 0.63 and 0.77.
  • The Analogy: This is like a weather forecast that is "mostly right." It's better than a coin flip, but it's not a guaranteed 100% prediction. It gives the librarian a strong hint, not a certainty.
• The Calibration: This is crucial. It means if the model says "You have a 70% chance of getting better," then out of 100 people with that score, about 70 actually did get better. The model wasn't lying or exaggerating; it was honest.

5. Who is Most at Risk? (The Key Predictors)

The model found that certain "red flags" made it harder for people to recover with standard therapy. These weren't just about how sad they felt, but about their life circumstances:

• Unemployment due to sickness: If you can't work because you're sick, it's harder to get better.
• Language barriers: Not speaking English well made a big difference.
• Financial stress: Receiving benefits or being unable to work.
• Physical health: Having other long-term conditions or disabilities.
• Ethnicity and Religion: The model showed that people from certain minority groups (like Black or Muslim communities) often had different outcomes, likely due to systemic barriers or lack of cultural fit in the therapy, not because of the therapy itself.

6. Why Does This Matter? (The "So What?")

This isn't about labeling people as "hopeless." It's about personalized care.

• The Old Way: Everyone gets the same standard therapy. If it doesn't work, the patient feels like a failure, and the therapist feels frustrated.
• The New Way (The Goal): The Crystal Ball rings a bell at the start.
  • Scenario A: "This patient has a high risk of not improving with standard therapy because they are unemployed and have high anxiety."
  • Action: Instead of just giving them the standard book, the librarian says, "Let's get you a job coach and a therapist," or "Let's try a different type of therapy right away."

The Bottom Line

This study is a blueprint for a smarter, fairer system. It shows that by looking at the whole person—not just their symptoms, but their job, their language, and their life struggles—we can predict who might struggle and give them extra support before they fall through the cracks.

The Catch: This "Crystal Ball" was tested in one specific city (South London). To make it work for the whole country, it needs to be tested in other places to ensure it works for everyone, everywhere. But it's a giant leap forward from guessing to knowing.

Technical Summary

1. Problem Statement

Depression and anxiety are highly prevalent in the UK, yet the NHS Talking Therapies (NHS-TT) programme, despite being the largest provider of evidence-based psychological treatments, achieves a recovery rate of only ~50%. A significant portion of patients do not respond to standard care.

• The Gap: Previous research has largely focused on group-level associations rather than individual-level prediction. Existing models often suffer from small sample sizes, lack of ethnic diversity (limiting generalizability), and poor methodological rigor (specifically regarding calibration and overfitting).
• The Goal: To develop and internally validate robust, multivariable prediction models using routinely collected data from a large, ethnically diverse sample. The aim is to enable fair, data-driven treatment stratification to identify patients at risk of poor outcomes early in the treatment pathway.

2. Methodology

Data Source and Sample

• Dataset: Extracted from the Clinical Record Interactive Search (CRIS) system of the South London and Maudsley (SLaM) NHS Foundation Trust.
• Timeframe: January 1, 2018, to August 27, 2024.
• Initial Pool: 155,945 patients.
• Final Analytical Sample: 30,999 adults (aged ≥18) who completed high-intensity therapy (e.g., CBT) consisting of 3–21 sessions.
• Demographics: Predominantly female (73%), median age 34. The sample is ethnically diverse: 57% White, 22% Black/Black British, and significant representation of other ethnic groups, addressing a common limitation in prior literature.

Outcomes (Binary Variables)
Seven distinct binary outcomes were modeled for depression and anxiety, plus functional impairment:

1. Reliable Improvement: Symptom reduction exceeding measurement error (PHQ-9 >6, GAD-7 >4).
2. Recovery: Moving from clinical caseness at baseline to non-caseness at endpoint.
3. Reliable Recovery: Meeting both criteria above.
4. Functional Impairment: Scoring ≥10 on the Work and Social Adjustment Scale (WSAS) at the end of treatment.

Predictors
A comprehensive set of baseline variables was used:

• Sociodemographic: Age, gender, ethnicity, religion, sexual orientation, English proficiency, Index of Multiple Deprivation (IMD), employment status, benefit receipt, and Statutory Sick Pay.
• Clinical: Baseline symptom severity (PHQ-9, GAD-7, WSAS), primary diagnosis, presence of long-term conditions/disabilities, psychotropic medication usage, and previous referrals.

Model Development & Validation

• Algorithm: Elastic Net Logistic Regression. This was chosen to balance bias and variance, reduce overfitting, and ensure interpretability while handling correlated predictors.
• Preprocessing: Missing values (≥70% completion threshold) were imputed using K-Nearest Neighbors (KNN) with $K=5$. Continuous variables were scaled/centered; categorical variables were dummy-coded.
• Hyperparameter Tuning: Grid search with 5-fold cross-validation to optimize the mixing parameter ($\alpha$) and regularization strength ($C$).
• Validation Strategy: Nested Bootstrap Resampling (200 iterations). This is a rigorous method that corrects for optimism (overfitting) by re-training and tuning the model within each bootstrap sample before evaluating on the original dataset.
• Metrics:
  • Discrimination: Area Under the Curve (AUC), Sensitivity, Specificity.
  • Calibration: Calibration intercept, slope, and LOWESS curves (assessing agreement between predicted probabilities and observed outcomes).
  • Stability: Assessed via prediction instability and Mean Absolute Prediction Error (MAPE).

3. Key Results

Model Performance
The Elastic Net models demonstrated moderate to good discrimination and strong calibration across all outcomes.

• AUC Ranges:
  • Depression Reliable Improvement: 0.76 (95% CI: 0.75–0.76).
  • Anxiety Reliable Improvement: 0.75 (95% CI: 0.74–0.75).
  • Functional Impairment: 0.77 (95% CI: 0.77–0.78).
  • Reliable Recovery (both depression and anxiety) showed lower performance (~0.63–0.67), likely due to the complexity of predicting the dual criteria of threshold crossing and symptom reduction.
• Calibration: Calibration slopes were close to 1.00 and intercepts near 0.00, indicating the models are well-calibrated and suitable for generating individual risk estimates.
• Comparison: The models outperformed or matched previous studies (e.g., Bone et al., 2021) and demonstrated superior calibration compared to other recent NHS-TT models that often omitted this metric.

Key Predictors (Feature Importance)
The most influential predictors aligned with clinical expectations but highlighted specific structural factors:

1. Unemployment due to sickness/disability: The strongest predictor across depression and anxiety outcomes.
2. Baseline Symptom Severity: PHQ-9 and GAD-7 scores were critical, though their predictive weight varied by outcome type.
3. Functional Impairment (WSAS): A top predictor for all outcomes, particularly for predicting post-treatment functional impairment.
4. Sociodemographic Factors:
   • Language: Inability to speak English.
   • Economic Status: Receipt of benefits and unemployment.
   • Ethnicity/Religion: Being Muslim and identifying as Black were associated with specific outcome risks.
   • Disability/Long-term Conditions: Presence of comorbidities.
5. Medication: Status of psychotropic medication usage (prescribed vs. not).

Sensitivity Analyses

• Alternative models (LASSO, Random Forest, Gradient Boosting) were tested. While Random Forest offered slightly better discrimination, it suffered from poor calibration. Elastic Net was retained for its balance of accuracy, calibration, and interpretability.
• Excluding data from the COVID-19 pandemic period (March 2020–Jan 2022) did not alter model performance, suggesting robustness to treatment practice shifts during the pandemic.

4. Key Contributions

1. Methodological Rigor: This study adheres to TRIPOD-AI guidelines, utilizing nested bootstrap resampling to provide optimism-corrected performance metrics, a standard often missing in previous mental health prediction literature.
2. Ethnic Diversity: By utilizing a sample where 43% identify as Black or Minority Ethnic (BME), the study addresses the "homogeneity bias" of previous models, ensuring the tools are tested in a population that is often underrepresented in research but overrepresented in South London services.
3. Calibration Focus: The study emphasizes that for clinical utility, models must be well-calibrated (not just discriminative). The reported models achieve this, making them safer for potential clinical decision support.
4. Granular Outcomes: Instead of pooling outcomes, the study models specific domains (depression vs. anxiety, improvement vs. recovery), acknowledging that patients may respond differently across these dimensions.

5. Significance and Implications

Clinical Application

• Stratified Care: These models can identify patients at high risk of non-response at the point of assessment. This allows clinicians to:
  • Adjust treatment intensity or modality earlier.
  • Initiate more frequent monitoring.
  • Integrate adjunctive support (e.g., employment advisors, peer support) for those with specific risk factors like unemployment or disability.
• Reducing Bias: The models highlight that factors like ethnicity, language, and employment status are strong predictors. This serves as a data-driven alert to clinicians and service managers that structural barriers (racism, poverty, lack of cultural competence) are actively driving poor outcomes, necessitating systemic changes rather than just clinical adjustments.

Policy and Future Work

• Equity: The findings underscore the need for the NHS to address social determinants of health (e.g., expanding employment support within therapy services) to improve mental health equity.
• Next Steps: The authors note that while internal validation is robust, external validation in independent datasets (other NHS trusts) is required before clinical implementation. Future work should focus on prospective trials to measure the "net benefit" of using these models to guide treatment decisions.

In summary, this paper provides a statistically robust, ethically conscious foundation for moving NHS Talking Therapies from a "one-size-fits-all" approach to a personalized, stratified care model, leveraging routine data to improve equity and outcomes for diverse patient populations.