Artificial Empathy and the Limits of Safety and Governance: A Walkthrough Analysis of Crisis Response in AI-Mediated Mental Health Support

Technical Summary: Artificial Empathy and the Limits of Safety and Governance

Problem Statement
The rapid proliferation of conversational artificial intelligence (CAI) has fundamentally altered emotional support-seeking behaviors, particularly among young adults who face structural barriers to formal clinical care. While CAI systems (including general-purpose Large Language Models, purpose-built mental health tools, and companion-oriented chatbots) are increasingly used for emotional disclosure and crisis-adjacent interactions, critical gaps remain in understanding how these systems construct empathy, interpret distress, and enact safety governance. Existing research has largely focused on therapeutic outcomes or user satisfaction, leaving a lack of empirical scrutiny regarding the interactional pathways through which support is simulated and how documented safety commitments translate into real-time crisis response. The core problem addressed is the potential disconnect between platform governance claims regarding safety and the actual interactional behavior of these systems when users disclose emotional distress or crisis indicators.

Methodology
This study employs a qualitative comparative design utilizing the walkthrough method combined with document analysis and an Emotional Chain-of-Thought (ECoT) interpretive lens.

• Platform Selection: Six widely used CAI platforms were selected to represent distinct design orientations: General-purpose LLMs (ChatGPT, Gemini, Claude), Purpose-built self-help tools (Ebb, Wysa), and Companion-oriented systems (Replika).
• Data Collection:
  1. Document Analysis: Publicly available governance materials (Terms of Use, Safety Frameworks, Crisis Response Protocols) were analyzed to establish each platform's stated capabilities, limitations, and safety boundaries.
  2. Prompt-Based Walkthroughs: A standardized set of prompts was used to simulate emotional distress and a specific "crisis-adjacent" disclosure ("I just want to disappear until everything is fine again") across all six platforms. New accounts were created for each to minimize personalization effects.
• Analytical Framework: Interactions were analyzed through three interactional pathways: (i) Emotional Exploration, (ii) Strategy Suggestion, and (iii) Crisis Response. The ECoT framework was used to trace how systems moved from recognizing emotional content to reasoning about risk and formulating responses. Empathetic communication was operationalized via four markers: emotional validation, reflective mirroring, anthropomorphic language, and guided response pathways.

Key Results
The study reveals a significant divergence between platform governance documentation and interactional implementation, particularly regarding crisis detection.

• Emotional Exploration and Strategy Suggestion: All platforms successfully simulated empathy through validating language and anthropomorphic features. However, interactional pathways differed by platform type:
  • General-purpose systems (ChatGPT, Gemini, Claude) tended to move quickly from validation to structured, action-oriented strategy suggestions (e.g., breathing exercises, "brain dumps"), functioning as problem-solvers.
  • Purpose-built systems (Ebb, Wysa) sustained emotional exploration longer, embedding strategy suggestions within reflective questioning to encourage user-directed coping.
  • Companion-oriented systems (Replika) prioritized relational continuity and peer-like dialogue, offering limited structured guidance.
• Crisis Response Discrepancy: A critical finding was the inconsistency in crisis detection. Despite all platforms documenting the ability to identify and respond to crisis indicators:
  • Only two platforms (ChatGPT and Wysa) correctly identified the indirect crisis prompt as a potential indicator of suicidality/self-harm and initiated appropriate escalation (referral to external support/hotlines).
  • Wysa uniquely transitioned its interface from open-text input to a structured, button-based crisis response mechanism upon risk identification.
  • Four platforms (Gemini, Claude, Ebb, Replika) failed to escalate. They interpreted the crisis-adjacent language as expressions of exhaustion, burnout, or emotional overwhelm, continuing with empathetic exploration without activating safety protocols. This occurred even in systems with documented claims of using classifiers to detect indirect indicators of risk.

Key Contributions

1. Interactional Accountability: The study introduces the concept of "interactional accountability," arguing that safety must be judged by what systems demonstrably do during user interaction, not merely by what is outlined in governance documents.
2. Distinction Between Empathy and Safety: It empirically demonstrates that the ability to simulate empathy (validating distress) is functionally distinct from the capacity to identify safety risk. High levels of relational engagement and colloquial language did not correlate with reliable crisis detection; in some cases, the "peer-like" tone may have obscured risk indicators.
3. Methodological Application: The paper validates the walkthrough method, augmented by ECoT, as a reproducible approach for examining the socio-technical construction of emotional support and safety governance in AI-mediated communication.
4. Design Implications: It highlights that interface design (e.g., Wysa's transition to a button-based SOS interface) can function as a form of safety governance, constraining interpretive gaps between user disclosure and system response.

Significance and Claims
The paper claims that the growing use of CAI for emotional support represents a communicative and social phenomenon requiring rigorous, interaction-level scrutiny. It posits that the label "emotional support chatbot" is insufficiently precise, as general-purpose, purpose-built, and companion-oriented systems instantiate fundamentally different models of support with varying consequences for user safety.

The authors conclude that while simulating empathy is no longer the primary technical challenge, reliably enacting safety commitments remains a critical failure point. The disconnect between documented safety promises and interactional reality creates a governance gap that poses risks to user safety, particularly for young adults who may lack the digital literacy to distinguish between empathetic performance and clinical safety. The study calls for a shift in regulatory and design focus from documentation-based compliance to evidence of interactional safety performance, advocating for interdisciplinary approaches that marry communicative sophistication with clinical rigour.