💬 NLP

Why Do Self-Harm Prediction Models Struggle to Generalise? Lexical and Semantic Variations in Emergency Department Triage Notes

This paper investigates why NLP models for predicting self-harm in emergency department triage notes struggle to generalize across institutions, revealing that while core clinical themes remain consistent, significant variations in lexical expression and feature importance between hospitals significantly reduce cross-site model performance.

Original authors: Liuliu Chen, Mike Conway, Jo Robinson, Vlada Rozova

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Liuliu Chen, Mike Conway, Jo Robinson, Vlada Rozova

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart detective (an AI computer program) trained to spot people in crisis who might hurt themselves. This detective was trained by reading thousands of short, messy notes written by doctors in one specific hospital (let's call it City Hospital). In City Hospital, the detective became a star, correctly identifying almost everyone in trouble.

But when the researchers sent this same detective to a different hospital in the countryside (Regional Hospital), the detective suddenly started missing cases and making mistakes. The question this paper asks is: "Why did our detective get confused when we moved it to a new neighborhood?"

Here is the breakdown of what they found, using simple analogies:

1. The "Accent" Problem (Lexical Differences)

Think of the two hospitals as two different towns where people speak the same language but with different accents and slang.

City Hospital notes often included words about "social stressors," "relationship breakups," or specific mental health terms.
Regional Hospital notes often mentioned "police," "being tied up," or specific local laws (like "Section 351").

Even though both towns were talking about the same core problem (people hurting themselves), they used different words to describe it. The detective was trained to listen for the "City accent." When it heard the "Regional accent," it didn't recognize the warning signs because the vocabulary was different.

2. The "Highlighter" Mismatch (Feature Importance)

The researchers looked at which words the computer thought were the most important "clues" (like a highlighter marking key text).

In City Hospital, the word "suicide" was a huge, flashing red flag.
In Regional Hospital, that same word was still there, but the computer didn't think it was quite as important as other words.

It's like if a teacher tells a student, "The word 'cat' is the most important word in this story." The student memorizes that. But then the student goes to a different school where the teacher says, "Actually, in this story, 'dog' is the most important word, and 'cat' is just a minor detail." The student gets confused because the rules for what counts as a "clue" changed, even though the story was about the same animal.

3. The "Same Story, Different Book" (Semantic Similarity)

The researchers used a special tool (called BERTopic) to look at the big picture of what the notes were about, ignoring the specific words used.

The Good News: The themes were almost identical. Both hospitals were mostly talking about people overdosing on pills, cutting themselves, or hanging. The "story" was the same.
The Bad News: The words used to tell that story were very different.

Imagine two people describing a car crash.

Person A says: "The vehicle collided with a barrier."
Person B says: "The car hit the fence."
The meaning is the same, but the words are different. The AI detective was so focused on the specific words ("vehicle" vs. "car") that it failed to realize both sentences meant the same thing.

4. Why the Detective Failed

The study concludes that the AI didn't fail because it didn't understand the concept of self-harm. It failed because it learned to rely on specific habits and writing styles unique to the first hospital.

City Hospital had a mental health team that wrote detailed notes about emotions and relationships.
Regional Hospital had different staff and different patient situations (more police involvement, different types of injuries), leading to different note-taking styles.

The AI learned the "handwriting" of the first hospital, not the universal "language" of self-harm.

What Can Be Done? (The Paper's Suggestions)

The authors suggest a few ways to fix the detective so it works in both towns:

Teach the detective the "meaning" instead of just the "words." Focus on the idea behind the sentence, not just the specific spelling.
Standardize the notes. If doctors wrote their notes in a more uniform way (like filling out a form instead of scribbling freehand), the AI wouldn't get confused by different slang or abbreviations.
Group similar words. Instead of treating "55mg" and "55 mg" as totally different clues, teach the AI to see them as the same thing.

The Bottom Line

The paper isn't saying the AI is useless; it's saying that AI models are currently too sensitive to local habits. Just like a person who only knows how to drive in a city might get lost in the countryside, this AI needs to learn the universal rules of self-harm detection, not just the specific way one hospital writes its notes.

Note: The authors emphasize that these tools are currently for public health monitoring (counting how many people are in trouble), not for making immediate medical decisions for individual patients.

Technical Summary: Lexical and Semantic Variations in Emergency Department Triage Notes for Self-Harm Prediction

Problem Statement
While Natural Language Processing (NLP) models have demonstrated robust performance in detecting self-harm presentations within single-hospital emergency department (ED) triage notes, their generalizability across different institutions remains a significant challenge. Previous studies (e.g., Rozova et al., 2022) achieved high performance metrics (e.g., AUPRC of 0.85) on test sets sharing the same distribution as training data. However, when a model trained on data from the Royal Melbourne Hospital (RMH) was applied to Latrobe Regional Health (LRH), a regional hospital serving rural communities, performance declined (AUPRC dropped from 0.85 to 0.78). The specific lexical, semantic, and structural factors contributing to this cross-site generalization gap in clinical text remain under-explored.

Methodology
The study conducted a comparative corpus analysis of ED triage notes from RMH and LRH collected between 2012 and 2017. The datasets were manually annotated by psychology experts as positive (self-harm) or negative (control). The analysis focused on three primary dimensions:

Lexical Characteristics: Descriptive statistics were computed for character and sentence counts to identify structural differences.
Feature Importance and Distribution:
- Feature Selection: Important features (unigrams, bigrams, trigrams) were identified independently for each hospital using Chi-Square tests (correlation with the target) and XGBoost (predictive contribution). The final set of "important features" was defined as the intersection of features selected by both methods.
- Comparative Analysis: The study compared the statistical association (Chi-Square) and predictive importance (XGBoost ranking) of shared features. It also computed TF-IDF representations using a shared vocabulary to compare document-level value distributions between sites.
- Longitudinal Analysis: Feature selection was repeated annually to assess the consistency of feature presence over time.
Topic Modelling: BERTopic was applied to the positive (self-harm) class of each dataset using 'all-mpnet-base-v2' sentence embeddings. Topic similarity was calculated using cosine similarity, with a threshold of 0.75 established via sensitivity analysis to identify unique topics. Jaccard similarity was also used to compare lexical overlap.

Key Results

Lexical and Feature Variations: While core themes (self-poisoning, self-injury) were consistent, the lexical expression varied significantly.
- Shared Features: Only a subset of features (e.g., "self harm," "od," "suicide attempt") were shared between hospitals. Even for these shared terms, their statistical association and predictive importance rankings differed between RMH and LRH.
- Distribution Shifts: Cosine similarity between site-level mean TF-IDF vectors was moderate for unigrams (0.822) but dropped significantly for bigrams (0.628) and trigrams (0.473). Most shared features showed statistically significant differences in TF-IDF distributions ( $p < .001$ ).
- Unique Features: LRH notes contained more terms related to police involvement (e.g., "with police," "section 351"), whereas RMH notes featured more social and psychological stressors (e.g., "relationship breakdown," "social stressors").
Longitudinal Consistency: A small core of features (20 unigrams, 9 bigrams, 2 trigrams) remained consistent across all years in both datasets, primarily related to overdose and self-inflicted harm. However, features related to specific medications and mental disorders showed higher variability between sites.
Semantic vs. Lexical Discrepancy: BERTopic identified 30 topics in LRH and 74 in RMH. The average semantic similarity across documents was 0.46, and topic representation similarity was 0.68, indicating high thematic consistency. However, the low Jaccard similarity revealed that while hospitals discuss similar topics, the specific lexicons used differ. Notably, 10 unique topics were identified in RMH (related to psychiatric complexity and specific overdoses), while no unique topics were found in LRH at the 0.75 threshold.

Significance and Contributions
The paper provides empirical evidence that the decline in model generalizability is driven by institutional variations in documentation style and patient population rather than fundamental differences in the nature of self-harm presentations.

Diagnosis of Generalization Failure: The study demonstrates that models trained on single-site data may inadvertently rely on site-specific documentation patterns (e.g., specific abbreviations, police involvement markers, or social stressor phrasing) rather than stable indicators of self-harm risk.
Thematic Consistency vs. Lexical Variability: The findings highlight a critical distinction: while the semantic themes of self-harm (overdosing, self-injury) are consistent across institutions, the lexical realization of these themes varies significantly.
Pathways for Improvement: The authors suggest that future efforts to improve cross-hospital model transferability should focus on normalizing text to reduce lexical variability. Proposed strategies include:
- Incorporating lexical semantics as predictive features.
- Standardizing alphanumeric combinations (e.g., drug dosages) into higher-level features.
- Normalizing triage notes into standardized formats to mitigate grammatical and lexical inconsistencies.

Limitations
The authors note that the study is limited to two Australian hospitals, which constrains the broader generalizability of the findings. Differences in dataset size and potential annotation discrepancies may influence feature representations. Furthermore, the study utilized TF-IDF and BERTopic for interpretability; the implications for contextual embedding models and Large Language Models (LLMs) remain an open question, though the authors hypothesize that models with greater capacity to generalize across surface-level lexical differences may partially mitigate the portability gap. The study emphasizes that these models are intended for public health surveillance rather than clinical decision-making.