💻 computer science

Uncovering Intervention Opportunities for Suicide Prevention with Language Model Assistants

This paper demonstrates that language models can effectively assist public health experts in analyzing sensitive suicide investigation records by matching existing annotations with high accuracy, surfacing discrepancies for review, and enabling efficient human-in-the-loop refinement of annotation guidelines for new variables.

Original authors: Jaspreet Ranjit, Hyundong J. Cho, Claire J. Smerdon, Yoonsoo Nam, Myles Phung, Jonathan May, John R. Blosnich, Swabha Swayamdipta

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jaspreet Ranjit, Hyundong J. Cho, Claire J. Smerdon, Yoonsoo Nam, Myles Phung, Jonathan May, John R. Blosnich, Swabha Swayamdipta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the National Violent Death Reporting System (NVDRS) as a massive, chaotic library containing over 270,000 tragic stories about suicide in the United States. Each story is a "narrative" written by investigators, but alongside these stories, there are also strict, pre-filled forms (structured data) where experts have manually checked off boxes like "Was the person depressed?" or "Did they have a gun problem?"

The problem is that filling out these forms is like trying to find a specific needle in a haystack while wearing heavy gloves. It's slow, emotionally draining, and sometimes the person filling out the form misses a detail in the story that contradicts what they checked off on the form.

This paper is about handing a super-smart digital assistant (a Language Model) to the human experts to help them sort through this library faster and more accurately.

Here is how the paper breaks it down, using simple analogies:

1. The "Second Pair of Eyes" (Checking Existing Data)

The researchers first asked: Can this AI assistant read the stories and fill out the forms just as well as the human experts?

The Test: They let the AI read 50 different types of "check-box" questions (like "Was there a school problem?") across thousands of stories.
The Result: The AI agreed with the human experts about 85% of the time.
The Hidden Gem: When the AI and the human disagreed, the researchers asked a senior expert to look at the case again. They found that 38% of the time, the human expert had actually made a mistake or missed a detail in the story that the AI caught.
- Analogy: It's like a spellchecker. If you type "their" instead of "there," you might not notice it, but the computer does. Here, the computer helped find errors in the human's notes that were hiding in plain sight.

2. The "Co-Pilot for New Ideas" (Inventing New Checkboxes)

The library's forms are old. They don't have checkboxes for everything that might be important today. For example, the forms didn't originally have a box for "Did the victim interact with a lawyer?"

Usually, creating a new checkbox is a nightmare. Experts have to:

Read hundreds of stories.
Write a rulebook (a "codebook") defining exactly what counts.
Have a team of people read thousands of stories to test the rulebook.
Repeat this for weeks.

The researchers built a human-in-the-loop algorithm (a "Co-Pilot" system) to speed this up:

Step 1: The AI guesses the answers based on a rough idea of the rule.
Step 2: The human expert only looks at the cases where the AI was wrong.
Step 3: The expert says, "No, that's not a lawyer interaction; that's just a police report," and explains why.
Step 4: The AI learns from that one correction and updates its rulebook instantly.
The Result: This process turned a task that usually takes weeks into something that took hours. The final rulebook created by this team-up was just as good as one created by humans working the old, slow way.

3. The Big Discovery: The "Lawyer" Connection

Using this new, fast method, the team looked for a specific pattern: Did suicide victims interact with legal professionals (lawyers, judges, police)?

The Finding: They found that in 12.7% of all the stories (over 100,000 cases), there was evidence of these interactions.
Why it matters: These interactions were often "implicit." For example, a story might say, "The victim was going through a messy divorce," without explicitly saying "they saw a lawyer." Humans might miss this as a "legal interaction," but the AI, guided by the new rulebook, caught it.
The Implication: This suggests that lawyers and legal professionals are a huge, untapped group of people who could help prevent suicide, but they aren't currently part of the standard prevention training.

The Golden Rules (What the Paper Actually Says)

The authors are very careful not to overpromise. They emphasize three key points:

AI is a Helper, Not a Replacement: The AI is a tool to make experts faster and catch their mistakes. It should never replace the human expert, especially in high-stakes situations like suicide prevention.
Safety First: The AI is great at spotting concrete things (like "gun" or "arrest") but struggles with vague feelings (like "depressed mood"). For those tricky emotional topics, humans must remain in charge.
Privacy: The data used was already stripped of names and personal details. The AI ran on local computers, so no sensitive data was sent to the internet.

In a nutshell: The paper shows that by pairing a smart AI with human experts, we can clean up old data, find hidden mistakes, and quickly discover new patterns (like the link between suicide and the legal system) that could save lives in the future.

Technical Summary: Uncovering Intervention Opportunities for Suicide Prevention with Language Model Assistants

Problem Statement

The National Violent Death Reporting System (NVDRS) is a critical public health resource containing structured and unstructured data on over 270,000 suicide cases in the United States. While the structured data is manually annotated by experts using extensive codebooks, the process is emotionally taxing, time-consuming, and prone to inconsistencies between the unstructured narratives (derived from coroner/medical examiner and law enforcement records) and the structured labels.

Currently, discovering novel suicide intervention opportunities requires experts to manually analyze narratives, develop new codebooks for unmeasured variables, and retroactively annotate the entire dataset. This manual bottleneck limits the ability to identify non-clinical risk factors and emerging intervention avenues. The paper investigates whether Large Language Models (LMs) can serve as effective assistants to alleviate this burden, surface data inconsistencies, and accelerate the development of codebooks for new variables.

Methodology

The research is structured around two primary research questions (RQs) and corresponding methodologies:

1. LM as an Annotation Assistant (RQ1)

The authors evaluate whether LMs can replicate human annotations for existing NVDRS variables and identify discrepancies between narratives and structured data.

Setup: They deployed locally hosted, open-weight LMs (including Llama-3-70B, Qwen2.5, and Mistral) as annotation assistants.
Prompting: Prompts were constructed by adapting official NVDRS codebooks (definitions, response options, discussion guidelines) into a standardized format. The models were tasked with generating Chain-of-Thought (CoT) reasoning and binary labels (0/1) for 50 existing variables.
Evaluation: Performance was measured against human abstractor annotations on a balanced dataset (500 narratives per variable) and a random sample reflecting real-world class imbalance.
Inconsistency Detection: For variables with low agreement, the authors conducted an expert review of cases where the LM and human abstractor disagreed to determine if the disagreement revealed genuine inconsistencies in the original data (narrative vs. structured label).

2. Human-in-the-Loop (HitL) Codebook Development (RQ2)

To address the lack of codebooks for new variables, the authors introduced an iterative algorithm to efficiently build codebooks.

Algorithm: The process begins with a coarse initial guideline ( $G_0$ ) containing only the variable name. The LM generates predictions on a sample of narratives.
Feedback Loop: An expert reviews only the incorrect LM predictions, providing the correct label and a rationale.
Refinement: The LM uses these expert corrections to synthesize and update the codebook ( $G_{t+1}$ ).
Termination: The loop continues until the LM achieves a target accuracy on a held-out validation set or a predefined annotation budget is reached.
Sampling: The algorithm employs coverage-based sampling to select narratives that are most dissimilar to previously annotated cases, ensuring diverse feedback.

3. Case Study: Legal Interactions

The HitL algorithm was applied to a novel variable: victim interactions with legal professionals. This variable was chosen because interactions with non-clinical professionals (e.g., lawyers) are under-explored in suicide prevention despite evidence of their relevance. The study compared the efficiency and quality of the HitL-generated codebook against a manually developed expert codebook.

Key Results

Annotation Performance

Agreement: Llama-3-70B achieved an average agreement of 85% with human abstractors across 50 variables on a balanced dataset.
Variable Specificity: Agreement was higher for concrete, firearm-related variables (e.g., GunPlaying, TPR > 0.96) and lower for mental health-related variables (e.g., SuicideThoughtHistory, TPR < 0.3).
Inconsistency Surfacing: In cases where the LM disagreed with the human abstractor, an expert review found that 38% of these instances represented genuine inconsistencies between the narrative and the structured data, compared to only 13% when they agreed. This suggests LMs can effectively flag data quality issues.

Codebook Development Efficiency

Simulation: In simulations using existing variables as "new" variables, the HitL-generated codebooks enabled LMs to achieve 80% agreement with human annotators, outperforming codebooks derived from official NVDRS guidelines (75% agreement).
Real-World Case Study:
- Efficiency: The HitL approach reduced codebook development time from weeks to hours (3.5 hours of expert time vs. ~4 weeks) and required annotating only 125 narratives (vs. 350) to achieve comparable performance.
- Quality: The HitL codebook achieved a Macro F1 score of 0.78, statistically on par with the manually developed expert codebook (0.78) across multiple models.
- Discovery: Applying the final HitL codebook to the full NVDRS dataset revealed that 12.7% of suicide narratives contain evidence of interactions with legal professionals (12.16% implicit, 0.53% explicit).

Significance and Claims

The paper claims that LMs can serve as effective, efficient assistants for public health researchers in high-stakes domains, provided they are used collaboratively with human experts rather than as replacements.

Efficiency in Data Analysis: LMs can significantly reduce the manual burden of annotating sensitive death narratives, achieving high agreement with human experts on structured variables.
Data Quality Improvement: LM predictions can surface inconsistencies between unstructured narratives and structured data, offering a mechanism to improve the quality of existing public health databases.
Accelerated Discovery: The proposed HitL algorithm enables the rapid development of codebooks for novel variables, allowing researchers to test new hypotheses (such as non-clinical intervention opportunities) without the prohibitive time costs of traditional manual coding.
New Intervention Avenues: The identification of frequent victim interactions with legal professionals suggests a substantial, previously unquantified opportunity for upstream suicide prevention interventions (e.g., training legal professionals to recognize risk).

Limitations and Ethical Considerations

The authors maintain a modest stance, acknowledging several limitations:

Scope: The evaluation is limited to a subset of 50 binary variables and a single case study (legal interactions); the algorithm's generalizability to other non-clinical factors or the full 600+ NVDRS variables remains to be fully tested.
Human Validation: Ground-truth validation for the entire dataset is infeasible due to the emotional toll on annotators; the study relies on a smaller expert-annotated test set.
Safety: The authors explicitly advise against relying on LMs as peer validators for low-agreement variables (particularly mental health topics) and recommend that models flag high-uncertainty predictions.
Human Oversight: Success depends on human-LM collaboration where LMs augment, not replace, domain expertise. The study emphasizes that codebook development must always be supervised by domain experts to mitigate safety risks.

The work concludes that while LMs offer powerful tools for data-driven discovery in suicide prevention, their deployment requires rigorous human oversight, ethical safeguards for annotator well-being, and a collaborative framework that prioritizes safety over coverage.