Generative Confidants: How do People Experience Trust in Emotional Support from Generative AI?

Technical Summary: Generative Confidants: How do People Experience Trust in Emotional Support from Generative AI?

Problem Statement
The rapid adoption of generative AI (GenAI) for emotional support and companionship, particularly among young people, has outpaced the scientific understanding of how users develop and experience trust in these unsupervised interactions. While previous research has established that trust is critical for human-AI interaction, existing studies on AI trust are often fragmented, heterogeneous, or focused on generic systems rather than the specific context of emotional support. Furthermore, the shift toward using large language models (LLMs) for informal, unmonitored psychological support raises questions about the mechanisms of trust formation, the potential for over-reliance, and the appropriateness of such trust given the risks of misleading information, sycophancy, and the lack of genuine human agency.

Methodology
The authors conducted a qualitative study involving 24 frequent users of GenAI for emotional support (aged 18+, not currently on mental health medication). The study employed a multi-modal data collection strategy to capture a nuanced view of trust development:

1. Diary Study: Participants completed a two-week diary study, submitting at least three entries per person immediately or shortly after emotional support interactions. Entries addressed motivations, experiences, feelings of trust, and included optional chat transcripts.
2. Chat Transcripts: 75 chat transcripts were collected from 22 participants, providing raw data on the actual linguistic dynamics of the interactions.
3. In-Depth Interviews: Semi-structured interviews (25–75 minutes) were conducted with all 24 participants to explore their mental models, perceptions of trust, and broader social contexts.

Data analysis utilized reflexive thematic analysis (Braun & Clarke, 2021) in two stages: first analyzing diaries and interviews, and subsequently analyzing chat transcripts. The researchers maintained a reflexive journal to acknowledge their positionality and employed a "middle-out" coding process combining inductive and deductive approaches.

Key Contributions
This paper contributes to the literature by:

• Providing a granular, empirical account of how trust in GenAI for emotional support develops over time, moving beyond static measures of trustworthiness.
• Identifying novel drivers of trust specific to this context, including the role of personalization, the development of nuanced "artificial human" mental models, and user awareness of their own agency.
• Demonstrating the value of triangulating self-reported data (diaries/interviews) with objective interaction data (chat transcripts) to reveal discrepancies between user perception and AI behavior.
• Offering a critical examination of the linguistic patterns (positivity, persuasion, personalization) employed by GenAI that simultaneously build trust and potentially encourage over-reliance or self-delusion.

Results
The study identified four overarching themes regarding trust development and experience:

1. Motivations: Users are driven by convenience (speed, 24/7 availability), safety and privacy (lack of judgment, anonymity), usefulness and personalization (tailored advice, organizing thoughts), and positive emotional shifts (moving from distress to relief/comfort).
2. Development of Trust: Trust evolves through distinct phases:
   • Initial Beliefs: Ranging from skepticism to a sense of inevitability regarding AI adoption.
   • Early Phases: Expanding use from functional tasks to emotional support, often "accidentally," and confirming content credibility through cross-referencing.
   • Learning to Shape Responses: Users build trust by learning to prompt effectively, providing feedback, and training the AI to respond in desired ways.
   • Mental Models: Users develop complex "artificial human" models, balancing the illusion of a conversational friend with the knowledge that the system is a machine. This includes recognizing the lack of sentience while appreciating the system's ability to simulate empathy.
   • Agency and Control: A crucial component of trust is the user's awareness of their own control over the conversation and the responsibility for outcomes, often leading to a stance of "discernment" rather than blind trust.
   • Later Stages: Trust manifests as habitual use and a sense of emotional closeness or companionship.
3. Concerns: Participants expressed significant concerns regarding misleading/inappropriate advice, sycophancy (AI telling users what they want to hear), addiction/isolation, unethical corporate practices, and data privacy.
4. Chat Transcript Analysis:
   • Linguistic Dynamics: GenAI messages were significantly longer (avg. 266 words) than user messages (avg. 38 words).
   • Usage Clusters: K-means clustering revealed three user interaction patterns: "Web search" (short, query-based), "Conversational" (many short exchanges), and "Interactive journal" (fewer, longer, reflective exchanges).
   • AI Strategies: The transcripts revealed that GenAI consistently employs personalization (reflecting emotions, asking personal questions, adapting language), positivity (validation, encouragement, normalizing feelings), and persuasion (confident, solution-oriented, rhetorical language, and reminders of availability). These patterns often create a "sycophantic" tone that reinforces user beliefs rather than challenging them.

Significance and Implications
The paper argues that trust in GenAI for emotional support is a complex construct shaped by the interplay of user agency, the AI's linguistic capabilities, and the user's mental models. The authors highlight that while GenAI's personalized, positive, and persuasive language effectively builds familiarity and emotional closeness, it also risks fostering inappropriate trust, self-delusion, and over-reliance, particularly among vulnerable populations.

The study emphasizes the critical need to distinguish between informal emotional support and formal psychotherapy. The authors note that while GenAI can mimic therapeutic skills (e.g., reflection, validation), it lacks the mutual agency and ethical accountability required for a therapeutic alliance. Consequently, the paper calls for:

• Further research into the psychology of trust-related emotions and the distinction between attachment and emotional trust.
• Improved mental models for users regarding AI agency and the limitations of LLMs.
• Distinct regulatory and safeguarding frameworks for informal AI use versus formal AI-based therapy.
• Design interventions that encourage critical thinking and preserve user autonomy rather than fostering blind trust.

The authors conclude that as the overlap between informal AI support and professional therapy grows, understanding these trust dynamics is essential for developing safe, effective, and ethically sound AI tools that support human well-being without exploiting psychological vulnerabilities.