Learning Evidence of Depression Symptoms via Prompt Induction

1. Problem Statement

The paper addresses the challenge of automatically identifying clinical evidence of depression symptoms in user-generated text (e.g., social media, forums) to support large-scale mental health screening.

• Task: Sentence-level classification of 21 specific depression symptoms defined by the BDI-II (Beck Depression Inventory-II) questionnaire.
• Data: The study uses BDI-Sen, a dataset annotated for first-person symptom evidence.
• Challenges:
  • Extreme Imbalance: Many symptoms have very few labeled positive examples.
  • Subtlety: Symptoms are linguistically subtle and often confused with "hard negatives" (sentences mentioning other symptoms or hypothetical/past states).
  • Limitations of Standard LLM Approaches: The authors find that common Large Language Model (LLM) strategies—Zero-Shot (ZS), In-Context Learning (ICL), and Supervised Fine-Tuning (SFT)—struggle to apply consistent, symptom-specific relevance criteria. ICL often over-predicts due to data sparsity, while SFT can overfit or fail to generalize on rare symptoms.

2. Methodology: Symptom Induction (SI)

The authors propose Symptom Induction (SI), a novel framework that transforms labeled training examples into concise, interpretable natural-language guidelines. Unlike ICL, which uses transient demonstrations, SI compresses these examples into reusable decision rules.

The SI Process:

1. Induction Phase:
   • For each symptom, a balanced demonstration set is constructed (up to 15 positive and 15 negative examples, including hard negatives).
   • A fixed "teacher" model (Gemma 3 27B) is prompted to generate a natural-language guideline for that specific symptom.
   • The guideline follows a structured format including: a core relevance question, patterns indicating presence, patterns that do not qualify (exclusion criteria), specific vocabulary, and edge cases.
2. Assessment Phase:
   • The induced guideline is injected into the system prompt of "student" models (smaller LLMs).
   • The model uses the guideline as a static decision rule to classify new sentences, rather than re-interpreting raw examples for every input.
   • Key Advantage: This decouples guideline generation from classification, allowing knowledge transfer across different model families without retraining.

3. Experimental Setup

• Datasets:
  • BDI-Sen: In-domain evaluation (1,516 sentences, 21 symptoms).
  • PsySym: Cross-domain evaluation (11,220 sentences, 7 disorders including Bipolar and Eating Disorders) to test generalization.
• Models: Eight LLMs from four families (Gemma 3, Llama 3, Phi-4, Qwen3) ranging from 3B to 14B parameters.
• Baselines: Zero-Shot Prompting (ZS), In-Context Learning (ICL), and Supervised Fine-Tuning (SFT).
• Metrics: Weighted F1-score (to account for class imbalance) and per-symptom F1.

4. Key Results

In-Domain Performance (BDI-Sen):

• Overall Performance: Symptom Induction (SI) achieved the best overall weighted F1 across six of the eight models tested.
  • Example: Gemma 3 4B with SI scored 0.389, significantly outperforming ZS (0.267), ICL (0.171), and SFT (0.242).
• Model Scale vs. Strategy: The choice of strategy (SI) had a larger impact than model size. A 4B model with SI outperformed larger models using other strategies.
• Error Analysis:
  • ICL suffered from severe over-prediction (high False Positives) due to difficulty distinguishing subtle negatives.
  • SFT was conservative but missed significant evidence (high False Negatives) on rare symptoms.
  • SI achieved the best balance, reducing both false positives and false negatives by providing explicit inclusion/exclusion criteria.
• Rare Symptoms: SI showed the most significant gains on infrequent symptoms (e.g., "Loss of Pleasure" improved from 0.224 F1 with ICL to 0.508 with SI). However, all methods failed on extremely sparse symptoms (e.g., "Loss of Interest in Sex" with only 1 positive test instance), highlighting data scarcity as the fundamental bottleneck.

Cross-Domain Generalization (PsySym):

• SI maintained the highest raw average F1 across Depression, Bipolar Disorder, and Eating Disorders.
• The performance gap between SI and ICL widened in cross-domain settings, suggesting that reusable induced guidelines generalize better than re-interpreting few-shot examples for new domains.
• Zero-shot prompting remained competitive for some symptoms where definitions alone were sufficient, but SI excelled where evidence was easily confounded.

5. Key Contributions

1. Symptom Induction (SI): Introduced a method to distill labeled examples into interpretable, reusable guidelines for symptom-level classification, framing the task as inducing relevance criteria.
2. Comprehensive Evaluation: Provided a rigorous comparison of inference strategies (ZS, ICL, SFT, SI) across multiple LLM families and datasets, identifying specific failure modes for each.
3. Per-Symptom Analysis: Detailed analysis showing that SI is most effective for subtle symptoms and hard negatives, while acknowledging the limits of current methods on extremely rare symptoms.
4. Cross-Disease Transfer: Demonstrated that guidelines induced for depression symptoms can effectively transfer to other disorders with shared symptomatology (Bipolar, Eating Disorders).

6. Significance and Limitations

Significance:

• Interpretability: SI provides explicit, human-readable criteria for why a sentence was classified as a symptom, addressing the "black box" issue of standard LLMs.
• Efficiency: It avoids the computational cost of fine-tuning and the token overhead of long-context ICL.
• Scalability: Offers a practical path for scaling mental health screening from user-generated text by leveraging small models guided by high-quality, induced rules.

Limitations:

• Teacher-Student Disparity: The current setup uses a larger teacher model (27B) to generate guidelines for smaller student models. It is unclear how much of the gain is due to the induction paradigm versus the teacher's superior capacity.
• Data Scarcity: The method cannot overcome the fundamental lack of data for extremely rare symptoms; it requires at least a few positive examples to induce meaningful criteria.
• Future Work: The authors suggest exploring clinician-in-the-loop refinement of guidelines and targeted data augmentation for underrepresented symptoms.