📄 medicine

Detecting Suicide Risk with AI Chatbots: Real-World Performance Within a Clinically Supervised Workflow

This retrospective cohort study demonstrates that a chatbot-integrated safety agent, operating within a clinically supervised workflow, achieved high sensitivity (93.2%) and specificity (99.7%) in detecting and escalating actionable suicide risk from real-world conversations, successfully identifying all clinician-rated immediate-risk cases.

Original authors: Emily J Ward, Kate H Bentley, Emily R Dworkin, Matt Hawrilenko, Emily Van Ark, John DeLorenzo, Millard Brown, Adam M Chekroud

Published 2026-10-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Emily J Ward, Kate H Bentley, Emily R Dworkin, Matt Hawrilenko, Emily Van Ark, John DeLorenzo, Millard Brown, Adam M Chekroud

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the digital age, people increasingly turn to artificial intelligence for help with their mental health, often finding a listening ear in chatbots that can be available at any hour. These tools are designed to offer support, but they face a critical challenge: knowing when a conversation has moved from general distress to a life-threatening emergency. Unlike a human therapist who can read tone, body language, and context, a computer program must rely solely on the words typed by the user to decide if someone is in immediate danger of harming themselves. The question of whether these machines can reliably spot the subtle signs of suicide risk is no longer just a technical puzzle; it is a matter of public safety. If a system fails to recognize a cry for help, the consequences are severe, yet if it raises the alarm too often for minor issues, it can overwhelm the human professionals who need to intervene.

To address this, researchers at Spring Health conducted a large-scale study to see how well a specific safety system works in the real world. They focused on a "guided intake" process, which is a brief, five-minute conversation a user has with a chatbot before meeting a human therapist for the first time. During this chat, the user shares what is troubling them, and a specialized safety agent—a piece of software designed to watch for danger—analyzes every message. This agent does not ask direct questions about suicide in every chat; instead, it listens for clues that the user might be thinking about ending their life, such as mentions of hopelessness, specific plans, or recent self-harm. When the agent detects a serious risk, it immediately stops the standard conversation and escalates the case, alerting a team of human clinicians who can reach out to the user right away.

The researchers wanted to know if this digital watchdog was accurate. They looked at more than 30,000 conversations from adults across the United States who used the service between August 2025 and June 2026. Because serious suicide risk is rare in the general population, simply looking at a random sample would not have provided enough data to judge the system's performance. Instead, the team used a careful sampling method to ensure they had enough examples of both low-risk and high-risk chats to make a fair assessment. They then brought in two licensed clinical psychologists to review a selection of these conversations. These experts read the transcripts without knowing what the safety agent had decided, acting as a gold standard to see if the machine was right. They rated each conversation on a scale ranging from no risk to immediate danger, just as the software did.

The results showed that the safety agent was highly effective at its job. When the human experts identified a conversation that contained actionable risk—meaning the user was expressing thoughts of suicide that required immediate attention—the agent correctly flagged and escalated 93.2 percent of those cases. Crucially, the system did not miss any of the most severe cases; not a single conversation that the human experts rated as an immediate emergency went undetected by the software. On the other side of the coin, when the experts determined that a conversation did not need to be escalated, the agent correctly held back 99.7 percent of the time. This means the system rarely sounded a false alarm, which is vital for keeping the human clinical team from being overwhelmed by unnecessary alerts. Of the conversations that the agent did escalate, about 81 percent were confirmed by the human experts as genuinely needing intervention.

The study also highlighted the efficiency of this approach. While a human clinician would take roughly a minute and a half to review a single transcript, costing about a dollar in labor, the automated safety agent performed the same screening for just four cents per interaction. Across the entire group of over 30,000 users, the total cost to run the automated check was approximately $1,200, whereas having a human review every single conversation would have cost between $28,000 and $35,000. This suggests that artificial intelligence can serve as a powerful, low-cost first line of defense, catching risks that might otherwise slip through the cracks. The researchers noted that in some instances, users disclosed dangerous thoughts to the chatbot that they had not admitted to in earlier written surveys, suggesting that people might feel safer sharing their darkest fears with a machine than with a person or a form.

However, the study also pointed out the limits of what this technology can do. The system was designed specifically to detect suicide risk and did not evaluate other dangers, such as threats of harm to others or severe symptoms like psychosis. Furthermore, the participants in this study were all adults, so the findings cannot be assumed to apply to teenagers or children, a group that is also increasingly using these tools. The researchers emphasized that while the detection system worked well, the ultimate goal is not just to find the risk but to ensure the response is helpful and safe. The study concludes that when integrated into a workflow where human clinicians are ready to step in, AI safety agents can provide a reliable, scalable way to protect users, offering a new layer of safety for those seeking mental health support.

Technical Summary: Detecting Suicide Risk with AI Chatbots in a Clinically Supervised Workflow

Problem Statement
The increasing deployment of generative AI (GenAI) chatbots for mental health support presents a critical public safety challenge: the ability of these systems to reliably detect and respond to suicide risk. While millions of individuals, including those with severe symptoms, utilize these tools, real-world evidence regarding the performance of AI-powered suicide-risk detection remains limited. Existing general-purpose chatbots often fail to distinguish between expressions of distress and indicators of imminent harm, potentially missing subtle risk signals or failing to provide clinically appropriate responses. Furthermore, evaluating these systems is methodologically difficult; clinically significant suicide risk is rare in routine use, making simple random sampling insufficient for estimating sensitivity. Conversely, restricting evaluation to already-escalated conversations introduces spectrum bias, preventing the assessment of specificity and false-positive burdens. The core problem addressed is the lack of empirical data on how well an AI safety agent performs in detecting spontaneously disclosed suicide risk within a naturalistic, real-world clinical workflow.

Methodology
This study employed a retrospective cohort design to evaluate a chatbot-integrated safety agent within Spring Health's "Guided Intake" workflow.

Data Source: The study analyzed 30,532 de-identified conversation transcripts from U.S. adults (aged 18–91) between August 20, 2025, and June 15, 2026. Participants were employees or dependents who completed self-report assessments (PHQ-9 and C-SSRS) and engaged in a five-minute AI-guided intake session prior to a therapy appointment.
The Safety Agent: The system utilized a Large Language Model (LLM, gpt-4o) powered safety agent. The agent was prompted with best-practice frameworks (e.g., C-SSRS, SAFE-T) to evaluate conversational content for intent, plan, means, timeframe, and recent behaviors. It classified conversations into four mutually exclusive severity levels:
1. No Risk: No evidence of suicidal thoughts or behavior.
2. Possible Risk: Ambiguous or indirect indicators (e.g., existential hopelessness, non-suicidal self-injury).
3. Non-Immediate Risk: Suicidal thoughts without intent, plan, or immediate timeframe.
4. Immediate Risk: Clear intent, plan, means, recent attempts, or short timeframe.
  Escalation Protocol: Conversations classified as Non-Immediate or Immediate Risk triggered an escalation protocol involving a safety message to the user, a ticket for the care team, and proactive outreach by 24/7 crisis clinicians.
Validation Strategy: To overcome the rarity of high-risk events, the researchers used stratified probability sampling. They drew a sample of 400 transcripts (450 total ratings) representing the full range of agent classifications and self-reported risk levels.
Clinical Review: Two blinded licensed clinical psychologists independently rated the transcripts using the same four-level scale. They were unaware of the agent's classification or the patient's self-reported assessment data. Disagreements were resolved by selecting the more severe rating.
Statistical Analysis: Primary metrics (sensitivity, specificity, positive predictive value) were calculated using inverse-probability weighting to recover estimates applicable to the original population of 30,532 conversations. Confidence intervals were derived via design-based stratified bootstrapping.

Key Contributions

Real-World Performance Metrics: The study provides rare, empirically grounded performance data for an AI safety agent in a live clinical setting, moving beyond theoretical or simulated evaluations.
Methodological Rigor: By employing stratified probability sampling and inverse-probability weighting, the study addresses the "spectrum bias" common in safety evaluations, allowing for robust estimation of both sensitivity (detecting risk) and specificity (avoiding false alarms) in a population where risk is rare.
Cost-Benefit Analysis: The research quantifies the economic feasibility of automated screening, comparing the marginal cost of the AI agent against human clinician review.
Benchmarking Against Human Consistency: The study includes a direct comparison of agent-clinician agreement against clinician-clinician agreement, establishing a human-level benchmark for the task.

Results

Sensitivity (Risk Detection): Among conversations clinicians judged to contain actionable suicide risk (Non-Immediate or Immediate), the safety agent correctly escalated 93.2% (95% CI: 88.8–97.8%).
Specificity (Avoiding False Alarms): Among conversations clinicians judged as not needing escalation, the agent correctly avoided escalating 99.7% (95% CI: 99.6–99.8%).
Positive Predictive Value (PPV): Of the conversations the agent escalated, 80.7% (95% CI: 74.3–86.5%) were confirmed by clinicians as genuinely warranting escalation.
Missed Risks: Crucially, zero conversations rated by clinicians as Immediate Risk were completely missed (classified as No Risk) by the agent. There were a small number of "undercalls" (7 conversations, weighted to ~27 population-wide) where the agent classified a Non-Immediate risk conversation as Possible Risk, failing to escalate it, though it did register some concern.
Agreement: Overall agreement between the agent and clinicians across the four-level scale was 95.5% (weighted $\kappa$ = 0.72). Inter-rater reliability between the two clinicians was high ( $\kappa$ = 0.91).
Cost Efficiency: The safety agent contributed approximately $0.04 per interaction. In contrast, clinician review of the same volume would have cost an estimated $28,000–$ 35,000 (approx. $0.92–$ 1.14 per transcript), representing a 96% cost reduction.
Discordance with Self-Report: The agent's risk classification showed low agreement with structured self-report assessments ( $\kappa \approx 0.09$ ). In approximately 5% of cases where participants denied suicidal ideation on assessments, the chatbot flagged at least possible risk, often due to spontaneous disclosures of family suicide, recent situational changes, or detailed mental health challenges not captured by the survey.

Significance and Claims
The paper claims that this study demonstrates the effective performance of an AI safety agent designed to detect and escalate suicide risk within a clinically supervised, real-world workflow. The authors assert that the agent successfully identified and escalated risk cases when risk was spontaneously communicated, with a critical finding that no clinician-confirmed immediate risk conversations went completely undetected.

The significance of these findings is framed around the necessity of "first, do no harm" in the era of GenAI mental health tools. The study suggests that AI safety agents can serve as a complementary safeguard, offering a scalable, cost-effective method for risk detection that operates alongside human clinicians. The authors position the agent not as a replacement for clinical assessment but as a tool that can catch risk disclosures that might otherwise go unspoken or unrecorded in standard assessments. They emphasize that while the agent performed well, it is part of a "human-in-the-loop" system where missed risks in the intake phase have a subsequent opportunity for detection by a therapist. The paper concludes that such AI-supported systems can achieve effective performance at scale, provided they are integrated into workflows with appropriate clinical oversight and ongoing evaluation.

Technical Summary: Detecting Suicide Risk with AI Chatbots in a Clinically Supervised Workflow

More like this