Development of a natural language processing application to extract and categorize mentions of violence from mental healthcare records text

This study developed and validated a multi-label BERT-based natural language processing application that successfully extracts and categorizes various forms of violence, patient roles, and contextual details from unstructured mental health records, achieving high performance on most features except temporal aspects.

The Gist

Imagine a massive library of millions of patient stories, written by doctors and nurses in a mental health hospital. These aren't neat spreadsheets with checkboxes; they are messy, handwritten-style notes full of sentences like, "The patient mentioned feeling threatened by their partner," or "History of physical altercations."

For years, researchers wanted to find specific stories about violence (like abuse, threats, or financial control) hidden inside these thousands of pages. But searching for them manually is like trying to find a specific grain of sand on a beach by looking at every single grain with a magnifying glass. It takes forever, and you might miss things.

This paper describes how the authors built a super-smart digital detective (an AI) to read these notes, find the violence, and sort it into neat categories automatically.

Here is the breakdown of what they did, using simple analogies:

1. The Problem: The "Needle in a Haystack"

Mental health records are full of valuable clues about violence, but they aren't organized. A doctor might write about a patient being abused, but they might also write about the patient threatening someone else, or just hearing about violence.

• The Challenge: The computer doesn't know the difference between "I was hit" (Victim), "I hit someone" (Perpetrator), or "I heard a fight" (Witness). It also doesn't know if the abuse happened yesterday or ten years ago, or if it was emotional (mean words) or financial (stealing money).

2. The Solution: Training a "Digital Intern"

The researchers decided to teach a computer program (called BERT, which is like a very advanced robot that has read the whole internet) how to understand these specific notes.

• The Training Camp: They couldn't just tell the robot what to do; they had to show it examples. They took 6,500 snippets of text from the hospital records.
• The Human Teachers: Two human experts read each snippet and labeled it like a teacher grading a test. They marked:
  • What kind of violence? (Emotional, Financial, Physical, Sexual).
  • Who was involved? (Was the patient the victim, the bully, or just watching?).
  • Is it real or a threat? (Did it actually happen, or was someone just saying "I might hit you"?).
  • When? (Did it happen recently or in the distant past?).
  • Where? (Was it at home or somewhere else?).

3. The "Magic" of the Multi-Label Model

Most old computer programs were like a one-question quiz: "Is this violence? Yes/No."
This new program is like a smart sorting machine. It looks at a sentence and asks many questions at once:

• "Is this emotional abuse?" (Yes)
• "Is the patient the victim?" (Yes)
• "Is it happening at home?" (Yes)
• "Is it a threat or an actual event?" (Actual)

By answering all these questions together, the AI understands the context much better than older tools.

4. The Results: A High-Scoring Detective

They tested their new AI on a "blind" set of notes it had never seen before. Here is how it did:

• The Superstars: It was amazing at spotting Emotional and Financial abuse (getting about 88-89% right). It was also great at figuring out if the patient was the victim or the bully, and if the violence was actually happening right now.
• The Struggles: It had a bit of trouble with Sexual violence (mostly because there were fewer examples to learn from) and Time.
  • Why the time trouble? Doctors often write in the past tense ("He was abused"). The AI sometimes got confused about when exactly it happened because the notes weren't specific enough. It's like trying to guess the exact date of a story just by reading "It happened a long time ago."

5. Why This Matters

Think of this AI as a search engine for human suffering.

• Before: Researchers had to read thousands of notes manually to study how violence affects mental health.
• Now: They can press a button, and the AI instantly pulls up every mention of financial abuse or domestic violence from millions of records.

This allows scientists to ask big questions, like: "Do people who suffer financial abuse have a higher risk of depression?" or "How does the role of a victim vs. a perpetrator change the treatment plan?"

The Bottom Line

The researchers built a powerful tool that can read messy, human-written medical notes and instantly understand the complex, painful stories of violence hidden inside. While it's not perfect yet (it still struggles a bit with timing), it's a giant leap forward in helping us understand the link between violence and mental health, potentially leading to better care for patients in the future.

Technical Summary

1. Problem Statement

Violence is a significant risk factor for mental health disorders and can also be a consequence of them. While Electronic Health Records (EHRs) contain rich narrative data regarding patient experiences, occurrences of violence (physical, sexual, emotional, financial) are rarely recorded as structured data. Instead, they exist within unstructured clinical free text.

• The Gap: Previous Natural Language Processing (NLP) efforts in this domain were limited. They often focused on single binary classifications (e.g., violence vs. no violence) or specific types (e.g., only physical/sexual), failing to capture the complexity of patient roles (victim vs. perpetrator), context (domestic vs. non-domestic), and the nature of the event (actual vs. threatened).
• The Challenge: Extracting nuanced information from clinical notes is difficult due to ambiguity, negation, hypothetical scenarios, and the need to distinguish between the patient's role and the setting of the violence.

2. Methodology

Data Source and Collection

• Dataset: The study utilized the Maudsley Clinical Record Interactive Search (CRIS) platform, which contains de-identified EHRs from the South London and Maudsley NHS Foundation Trust (serving ~1.3 million residents).
• Initial Retrieval: Two stages of keyword-based retrieval were performed:
  1. General/Physical/Sexual: Keywords included "assault," "rape," "hit," "violence," etc. (3,771 instances).
  2. Emotional/Financial: Keywords included "coercive," "gaslight," "financial abuse," "economic abuse," etc. (3,500 instances).
• Final Dataset: After deduplication and filtering, 6,685 text segments were identified. Each segment consisted of a 400-character window (approx. 200 characters before and after the keyword).

Annotation Process

• Guidelines: A comprehensive coding book was developed defining:
  • Violence Types: Emotional, Financial, Physical (non-sexual), Sexual, Unspecified, and Irrelevant.
  • Patient Roles: Victim, Perpetrator, Witness, Unclear, or combinations.
  • Features: Polarity (Affirmed, Abstract, Negated), Presence (Actual, Threat, Unclear), Temporality (Past, Not-past), and Setting (Domestic, Not domestic, Unclear).
• Human Annotation: Two raters independently annotated the data using Label Studio.
  • Pilot: 185 instances.
  • Double-Annotation: 700 instances (643 successfully resolved) to establish Inter-Annotator Agreement (IAA).
  • Single-Annotation with Overlap: The remaining ~5,800 instances were single-annotated with a 20% overlap for quality control.
  • Total Annotated: 6,500 instances were finalized for model development.
• Agreement: Cohen's Kappa values ranged from 0.70 to 0.88, indicating substantial agreement, with the highest agreement on Violence Type (0.88) and lowest on Temporality (0.75).

Model Architecture

• Algorithm: A Multi-label BERT model (Bidirectional Encoder Representations from Transformers) was employed. This approach was chosen over independent binary classifiers to capture dependencies and correlations between labels (e.g., a "victim" role often correlates with "domestic" settings).
• Training Strategy:
  • Split: 90% of data (5,849 instances) for training/validation; 10% (651 instances) held out as a blind test set.
  • Balancing: Training and test sets were stratified by gender, ethnicity, and age to ensure representativeness.
  • Validation: 5-fold cross-validation was used during training.
  • Hyperparameters: BERT tokenizer, max_length=256, 4 epochs.
• Task: The model was trained to simultaneously predict multiple attributes for each text segment: Violence Type, Patient Role, Polarity, Presence, Temporality, and Setting.

3. Key Results

The model was evaluated on the blind test set (651 instances) using Precision, Recall, and F1-scores.

High-Performance Categories

• Violence Types:
  • Emotional Violence: F1 = 0.89
  • Financial Violence: F1 = 0.88
  • Physical (Non-sexual) Violence: F1 = 0.84
  • Unspecified Violence: F1 = 0.81
  • Note: Sexual violence had a lower F1 (0.68) due to low recall (0.55), likely due to data scarcity.
• Patient Role:
  • Perpetrator: F1 = 0.89
  • Victim: F1 = 0.84
  • Limitation: "Witness" and combined roles performed poorly (F1 < 0.05) due to insufficient training examples.
• Contextual Features:
  • Presence (Actual Violence): F1 = 0.95 (Excellent performance).
  • Polarity (Affirmed): F1 = 0.89.
  • Setting (Domestic): F1 = 0.88.

Low-Performance Categories

• Temporality: The model struggled to distinguish between "Past" and "Not-past" violence.
  • F1 = 0.65 for "Past" violence.
  • Reason: Clinical notes are often written retrospectively, making tense usage ambiguous; inter-rater agreement for this category was also the lowest.
• Sexual Violence: Lower recall (0.55) suggests the model missed many instances, likely due to the small sample size (29 in test set).

4. Key Contributions

1. Multi-Label Capability: Unlike previous studies using single binary classifiers, this work successfully implemented a multi-label BERT model that captures the complex interdependencies between violence types, roles, and contexts.
2. Expanded Scope: The application moves beyond physical/sexual violence to include emotional and financial abuse, which are often overlooked in structured data but prevalent in mental health records.
3. Granular Characterization: The model extracts specific features previously unattainable at scale, such as distinguishing between victimization and perpetration, identifying domestic settings, and determining polarity (affirmed vs. abstract/threatened).
4. Scalable Tool for Research: The developed tool enables large-scale surveillance and research into the relationship between diverse forms of violence and mental health outcomes using routine clinical data.

5. Significance and Limitations

Significance:

• Research Impact: This tool allows researchers to move from small-scale manual chart reviews to large-scale epidemiological studies on violence exposure in mental health populations.
• Clinical Insight: By identifying patterns (e.g., female victims of frequent emotional abuse having higher depression risks), the tool can inform better routine documentation and clinical interventions.
• Methodological Advancement: Demonstrates the efficacy of transfer learning (BERT) in the specific domain of mental health NLP for complex, multi-dimensional extraction tasks.

Limitations:

• Temporality: The model cannot reliably determine when violence occurred (past vs. recent), a critical factor for clinical risk assessment.
• Data Scarcity: Rare categories (Witness, Sexual violence, Co-occurring violence types) suffered from low performance due to insufficient training data.
• Truth vs. Record: The model annotates what is written in the record, not the truth. It does not distinguish between delusional experiences of violence and actual events, which is a crucial distinction for clinical application.
• Generalizability: The model was trained on data from a specific UK mental health trust; performance may vary in other healthcare systems or languages.

Conclusion:
The study successfully developed a robust NLP application that significantly improves the extraction of violence-related data from unstructured mental health records. While challenges remain regarding temporal detection and rare categories, the tool offers a powerful new capability for understanding the intersection of violence and mental health.