Keyphrase Generative Representation of Youth Crisis Conversations Beyond Static Taxonomies

Technical Summary: Keyphrase Generative Representation of Youth Crisis Conversations Beyond Static Taxonomies

Problem Statement

Crisis Responders (CRs) at youth support services, such as Kids Help Phone (KHP), must rapidly assess thousands of unstructured SMS conversations to identify mental health concerns. Currently, these services rely on fixed-label taxonomies (e.g., anxiety, self-harm) to map conversations to predefined categories. While these static taxonomies offer consistency and auditability, they fail to capture the evolving, culturally grounded, and context-specific language used by youth in distress. Compressing complex narratives into a limited set of labels risks rendering important contextual signals invisible, potentially hindering escalation reviews, equity monitoring, and the identification of emerging mental health trends. Furthermore, maintaining exhaustively updated label systems is difficult as sociocultural contexts shift. While Large Language Models (LLMs) offer generative capabilities to surface nuanced themes, unconstrained outputs raise concerns regarding hallucination, variable specificity, and a lack of transparency required for real-world clinical deployment.

Methodology

The study utilized a dataset of 703,975 de-identified youth crisis conversations collected by KHP between 2018 and 2023. The research employed a hybrid framework combining structured classification with constrained generative representation:

1. Taxonomy Expansion: The legacy 19-label issue taxonomy was expanded to a 39-label hierarchical schema covering four domains: Health & Wellbeing, Safety & Violence, Identity & Relationships, and Other. This expansion was developed through iterative consultation with clinical experts and frontline CRs.
2. Multi-Label Classification: A zero-shot configuration of the Llama 3 8B model was used to assign labels from the expanded 39-label taxonomy. Model predictions were evaluated against expert annotations using three aggregation schemes: "Any" (selected by at least one annotator), "Top-2 Majority," and "Top-2 Consensus."
3. Keyphrase Generative Representation (KGR): To address the limitations of static labels, the authors introduced KGR, a constrained LLM-based framework. The system prompts Llama 3 to generate up to five concise, conversation-specific keyphrases that describe the main issues, emerging themes, and identity-linked experiences without producing diagnostic labels.
   • Constraints: Generation is restricted to brief, non-diagnostic phrases aligned with youth vernacular.
   • Evaluation: KGR outputs were evaluated via two methods:
     • Expert Evaluation: 129 conversations were annotated by three trained CRs (387 total annotations) to assess accuracy, relevance to lived experience, and ability to capture subtle details.
     • Automated Evaluation: Generated keyphrases were mapped to the legacy 19-label taxonomy using exact matching, sublabel matching, and embedding-based similarity (using all-MiniLM-L6-v2) to measure alignment with structured categories.
4. Operational Workflow Testing: The study compared a KGR-based topic retrieval workflow against a manual analyst workflow using keyword searches for specific topics (bullying and body image). KGR retrieval utilized cosine similarity between target topics and generated keyphrases, supplemented by Chain-of-Thought (CoT) prompting to extract verbatim supporting excerpts.

Key Results

• Taxonomy Expansion: The expanded 39-label taxonomy achieved high expert-consensus reliability. Under the "Top-2 Consensus" aggregation scheme, the model achieved an accuracy of 0.96 and an AUROC of 0.82, outperforming other aggregation schemes.
• KGR Accuracy and Utility: Expert review indicated that 81% of generated keyphrases accurately reflected conversation content. Additionally, 74% of experts agreed that the keyphrases improved clarity, and 71% found them useful for understanding conversations.
• Emergent Themes: KGR successfully surfaced identity-linked and situational themes absent from the fixed taxonomy, including immigration problems, caregiver burden, Indigenous community contexts, residential school survivorship, and microaggressions.
• Topic Retrieval Performance: In operational testing, KGR-based retrieval significantly outperformed manual analyst workflows. For bullying-related conversations, KGR achieved 70% accuracy compared to 25% for manual retrieval (a 45-percentage-point improvement). For body image topics, KGR achieved 73% accuracy versus 53% for analysts. The improvement was driven by higher precision, as KGR avoided the false positives common in keyword-based searches while maintaining 100% recall.
• Category Variance: Agreement was highest for high-salience categories (e.g., Racism, Ableism, Suicide) and lower for service-state or interactional categories (e.g., Prank, Testing Service), suggesting generative representations are most effective for explicit semantic signals of distress.

Key Contributions

1. Dataset and Taxonomy Expansion: The authors evaluated a national-scale dataset of over 700,000 conversations and demonstrated that expanding the issue taxonomy from 19 to 39 labels improves topic coverage and structured issue identification, enabling a broader representation of youth mental health and identity-related concerns.
2. Keyphrase Generative Representation (KGR): The study introduces KGR, a constrained LLM framework that augments static labels with concise, non-diagnostic keyphrases. This approach was shown to accurately reflect conversation content while revealing new contextual and identity-linked patterns that fixed labels miss.
3. Operational Evidence for Hybrid Systems: The paper provides evidence that KGR can surface emerging, culturally grounded concerns (e.g., immigration challenges, caregiver burden) and supports a topic-retrieval workflow that significantly outperforms manual analyst processes in accuracy.

Significance and Claims

The paper posits that KGR marks a shift toward hybrid, interpretable generative representations that extend crisis response beyond static taxonomies. The authors claim that while high classification accuracy within a label space is valuable, it does not guarantee the adequacy of the label space itself for capturing the diversity and evolution of youth distress.

The significance of this work lies in its demonstration that generative methods, when appropriately constrained and anchored within established taxonomic frameworks, can function as complementary layers. These layers enrich post-conversation representations without displacing established classification systems, thereby preserving operational continuity and auditability while surfacing contextual signals that would otherwise remain latent. The study argues that this hybrid approach offers a pragmatic pathway for integrating generative AI into mental health analytics, balancing the need for reliability and transparency with the capacity to capture nuanced, culturally grounded patterns of distress.