Finetuning Large Language Models for Automated Depression Screening in Nigerian Pidgin English: GENSCORE Pilot Study

1. Problem Statement

The study addresses the critical gap in mental healthcare access in Nigeria, specifically for depression screening. Key challenges include:

• Workforce Shortage: Nigeria has fewer than 300 psychiatrists for a population of over 200 million.
• Cultural and Linguistic Barriers: Standard screening tools like the Patient Health Questionnaire-9 (PHQ-9) are validated in high-income countries and English. They often fail to capture the nuances of Nigerian Pidgin, a lingua franca used by millions, which relies heavily on idioms, metaphors, and slang (e.g., "my head dey heavy" for fatigue) that standard models misinterpret.
• AI Limitations: Existing Large Language Models (LLMs) are trained primarily on high-resource languages and struggle with low-resource African dialects, leading to potential misclassification of symptoms or unsafe clinical advice.
• Safety Risks: General-purpose LLMs may hallucinate diagnoses or fail to escalate crisis situations (e.g., self-harm), posing significant risks in mental health applications.

2. Methodology

The study, titled GENSCORE, developed an end-to-end pipeline to adapt LLMs for automated, culturally safe depression screening in Nigerian Pidgin.

A. Data Collection and Curation

• Platform: Data was collected via GENCURATE, a custom platform.
• Dataset: 432 audio responses from Nigerian young adults (aged 18–40) were transcribed.
• Content: Responses were spontaneous answers to prompts mapped to the 9 items of the PHQ-9 (e.g., sleep patterns, eating habits).
• Split: The dataset was divided into Training (345), Validation (44), and Test (42) sets.

B. Preprocessing and Annotation

• Cleaning: Orthographic variations in Pidgin spelling were normalized.
• Multidisciplinary Annotation: A team of computational linguists, psychologists, and native speakers annotated the data on three dimensions:
  1. PHQ-9 Severity: Categorized into scores 0–3 based on clinical guidelines.
  2. Semantic Intent: Distinguished between literal and metaphorical/idiomatic expressions.
  3. Idiomatic Mapping: Mapped slang (e.g., "no get ginger") to clinical concepts (e.g., anhedonia/low energy).

C. Model Selection and Fine-Tuning

Three LLMs representing different scales were selected for Supervised Fine-Tuning (SFT):

1. Phi-3-mini-4k-instruct: A lightweight model (baseline for low-resource efficiency).
2. Gemma-3-4B-it: A mid-scale model with broader multilingual pretraining.
3. GPT-4.1: A large-scale model (upper-bound) fine-tuned via the OpenAI API.

Training Configuration:

• Format: Structured instruction-response pairs containing the Pidgin-adapted question, user response, and annotations.
• Hyperparameters: 4 epochs, batch size 8, learning rate 2e-5, temperature 0.4 (for open-source models).
• Safety Alignment: All models were constrained via Prompt Engineering to:
  • Adopt a "warm, respectful, empathetic" tone.
  • Avoid making diagnostic claims.
  • Strictly escalate self-harm disclosures (PHQ-9 Item 9).

D. Evaluation Framework

Performance was assessed using:

• Quantitative Metrics: Accuracy, Precision, F1-score for severity classification; BERTScore for semantic alignment; Safety compliance rates.
• Qualitative Metrics: Expert review by psychologists and native speakers evaluating clarity, relevance, and cultural appropriateness.

3. Key Results

Quantitative Performance

GPT-4.1 significantly outperformed the other models across all metrics:

• PHQ-9 Severity Accuracy: GPT-4.1 achieved 94.5% (vs. 82.7% for Gemma and 71.8% for Phi-3).
• Semantic Alignment (BERTScore): GPT-4.1 scored 0.95, demonstrating superior ability to interpret idioms and metaphors compared to Gemma (0.85) and Phi-3 (0.72).
• Safety Compliance: GPT-4.1 achieved 100% correct escalation for self-harm content and 99% non-diagnostic compliance.

Qualitative Performance

Human evaluators rated GPT-4.1 highest in:

• Cultural Appropriateness: 98% (vs. 82% for Gemma).
• Clarity and Relevance: 98% and 96% respectively.

4. Key Contributions

1. First Pidgin-Specific Mental Health Dataset: Creation of a curated, annotated dataset of 432 Nigerian Pidgin responses mapped to PHQ-9 items, addressing a major data scarcity in African NLP.
2. Cultural Adaptation Framework: Demonstrated that fine-tuning LLMs on culturally annotated data (handling slang, idioms, and metaphors) is essential for accurate symptom detection in low-resource settings.
3. Safety-First AI Architecture: Established a protocol combining SFT with strict prompt engineering to ensure models do not diagnose but provide supportive screening and crisis escalation.
4. Model Scaling Analysis: Provided empirical evidence that while smaller models (Phi-3) offer a baseline, larger models (GPT-4.1) are currently necessary for the high-level semantic understanding required for complex, idiomatic mental health conversations.

5. Significance and Future Work

• Scalability: The study proves that AI-mediated screening can be made safe, accessible, and linguistically appropriate for underserved Nigerian communities, potentially bridging the gap caused by the shortage of psychiatrists.
• Low-Resource Paradigm: It highlights that "low-resource" languages require not just translation, but deep cultural and semantic adaptation of AI models.
• Future Directions: The authors plan to expand the dataset for better generalizability, integrate the system into real-world digital health platforms, and extend the approach to other Nigerian languages and dialects.

Conclusion: The GENSCORE pilot successfully demonstrates that fine-tuned LLMs, when grounded in culturally specific data and safety protocols, can serve as effective, scalable tools for preliminary depression screening in linguistically diverse, resource-constrained environments.