Evidence-guided AI regularization for suicidal ideation prediction in pediatric bipolar disorder

Technical Summary: Evidence-guided AI regularization for suicidal ideation prediction in pediatric bipolar disorder

Problem Statement
Suicide prediction models in psychiatry frequently rely on purely data-driven feature selection. In modest-sized clinical samples, this approach often yields unstable predictor sets and clinically opaque models that fail to integrate established domain knowledge. While machine learning (specifically LASSO) offers a solution for the "small-n-large-p" problem common in pediatric psychiatry, standard LASSO applies a uniform penalty to all coefficients. This treats every candidate predictor as equally likely to be irrelevant before data observation, ignoring prior clinical evidence. Consequently, models may lock onto unstable, sample-specific correlations rather than clinically plausible risk domains. Furthermore, manually assigning feature-specific penalty weights based on expert opinion is often infeasible, non-reproducible, and difficult to scale.

Methodology
The authors developed Evidence-Based AI LASSO (EBAL), a hybrid framework that integrates curated clinical evidence directly into the regularization process of a LASSO model. The pipeline consists of four main stages:

1. Evidence Synthesis: A systematic review was conducted following Cochrane and PRISMA guidelines to construct a domain knowledge corpus. The search was restricted to higher-level evidence (systematic reviews, meta-analyses, network meta-analyses, umbrella reviews, and clinical practice guidelines) relevant to suicidality in youth with bipolar disorder or related domains. This process yielded 40 curated documents.
2. Retrieval-Augmented Generation (RAG) for Weighting: To translate the unstructured evidence corpus into numerical penalty weights, the authors employed a two-model Large Language Model (LLM) pipeline using open-weight Qwen3 models:
   • Qwen3-Embedding-8B: Used to convert the 40 evidence documents into high-dimensional vector representations and index them in a local ChromaDB vector database.
   • Qwen3-30B-A3B-Thinking: A mixture-of-experts reasoning model used to assign feature-specific penalty factors. For each of the 20 candidate predictors, the system retrieved the three most semantically relevant text segments from the evidence corpus. The LLM then evaluated these segments against a prespecified scoring rubric (scores 2–5) to determine the penalty factor. A score of 2 indicated strong evidence linking the predictor to suicidal ideation (lower penalty), while 5 indicated weak or no evidence (higher penalty).
   • Constraint: The LLM received only predictor names and retrieved evidence segments, with no access to the clinical outcome data, ensuring the weights were derived solely from prior knowledge.
3. Model Fitting: The derived penalty weights were used to fit a weighted LASSO logistic regression model. The study utilized baseline data from 136 youth with confirmed bipolar spectrum disorder from the Greater Houston Area Bipolar Registry. The primary outcome was suicidal ideation (binary), and the secondary outcome was suicidal behavior.
4. Evaluation: Performance was assessed using nested leave-one-out cross-validation (LOOCV). The inner loop selected the optimal regularization strength ($\lambda$) via 10-fold cross-validation, while the outer loop held out one participant for testing. The EBAL model was compared against a standard, evidence-agnostic LASSO with uniform penalties. Synthetic Minority Over-sampling Technique (SMOTE) was applied only to training data to address class imbalance.

Key Results

• Suicidal Ideation Prediction:
  • EBAL Performance: Achieved an AUROC of 0.768 and a balanced accuracy of 0.757.
  • Standard LASSO Performance: Achieved an AUROC of 0.760 and a balanced accuracy of 0.715.
  • Statistical Significance: While the difference in AUROC was not statistically significant ($p=0.313$), paired bootstrap tests confirmed that EBAL significantly outperformed standard LASSO in balanced accuracy (+0.042, $p=0.010$) and Matthews correlation coefficient (+0.079, $p=0.010$).
  • Feature Selection: EBAL produced a sparser, more stable model, retaining only 11 of 20 predictors compared to 18 for the standard LASSO. The top three positive predictors retained by EBAL were current depressed mood, duration of mood disorder illness, and comorbid generalized anxiety disorder.
• Suicidal Behavior Prediction:
  • Both models performed near chance (EBAL AUROC: 0.594; Standard LASSO AUROC: 0.592) and failed to achieve meaningful variable selection (both retained all 20 features). The authors attribute this to the acute, state-dependent nature of suicidal behavior, which is not well-captured by the static clinical predictors available in the dataset.

Significance and Claims
The paper claims that EBAL successfully operationalizes the principles of evidence-based medicine within the penalty structure of regularized regression. By embedding curated clinical evidence into the model before estimation, the framework produces a predictor set that is:

1. Clinically Coherent: The retained features align with established mechanistic pathways (e.g., internalizing symptoms, illness chronicity) rather than sample-specific noise.
2. Sparse and Stable: The evidence-guided penalties effectively filter out weak predictors, reducing model complexity without sacrificing (and slightly improving) classification performance.
3. Transparent and Reproducible: The use of open-weight LLMs and a traceable evidence corpus allows for full inspection of the penalty derivation process. Unlike proprietary cloud-based models, the open-weight approach ensures that the exact penalty weights can be reproduced by other groups without dependency on external APIs or changing model parameters.

The authors conclude that while EBAL did not improve the prediction of suicidal behavior (likely due to data limitations regarding acute triggers), it offers a robust, transparent strategy for aligning psychiatric prediction models with prior clinical knowledge, particularly for outcomes like suicidal ideation where risk factors are more stable and well-documented. The study acknowledges limitations including its cross-sectional design, single-site nature, modest sample size, and the lack of an external validation cohort.