💻 computer science

"Are you an AI?" Analyzing Client Suspicion of AI Use in Crisis Counseling

This study analyzes nearly 76,000 human-staffed crisis counseling conversations in India to reveal that client suspicion of AI use is rising, often triggered early in interactions and exacerbated by explicit denials, highlighting the need for careful AI integration to preserve the therapeutic relationship.

Original authors: Shreya Shah, Akshay Swaminathan, Meghana Simhadri, Ivan Lopez, Sharang Phadke, Divyanjali Verma, Abhay John, Luke Zhao, Fiona Cai, Sharon Zhang, Gloria Ye, Ivy Pham, William Wang, Sebastian Garcia, Sa

Published 2026-06-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shreya Shah, Akshay Swaminathan, Meghana Simhadri, Ivan Lopez, Sharang Phadke, Divyanjali Verma, Abhay John, Luke Zhao, Fiona Cai, Sharon Zhang, Gloria Ye, Ivy Pham, William Wang, Sebastian Garcia, Sarah Wornow, Angelina Wang, Nigam H. Shah

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a crisis counseling hotline as a digital lifeboat. When people are in deep emotional trouble, they jump into this boat to talk to a trained human crew member who can help them stay afloat.

Recently, there's been a lot of buzz about AI robots taking over jobs, including helping people with their feelings. This has made some passengers on the lifeboat suspicious. They start wondering: "Wait, is the person helping me actually a human, or is it a robot pretending to be one?"

This paper is a report from researchers who looked at 75,777 conversations on a real WhatsApp helpline in India to see how often this suspicion happens, what triggers it, and how the human counselors handle it.

Here is the breakdown of their findings, using simple analogies:

1. The "Robot Radar" is Getting Stronger

As more people use AI chatbots in their daily lives, they are getting better at spotting them. The researchers found that the number of people asking, "Are you a bot?" is rising.

The Trend: In June 2024, only 0.8% of people asked. By March 2025, that number jumped to 2.6%.
Who is asking? It's mostly young adults (ages 18–24) and men who are more likely to suspect they are talking to a machine.

2. Why Did They Get Suspicious? (The "Uncanny Valley" Effect)

Imagine you are talking to a friend, but they keep repeating the same phrase, take a long time to reply, or sound like a robot reading a script. You'd start to think, "Is this actually my friend?"

The study found that clients suspected AI when the counselor's behavior felt "off." The top triggers were:

The "Broken Record": The counselor gave repetitive or formulaic answers.
The "Ghost": The counselor didn't answer a specific question the client asked.
The "Wait Time": The counselor took too long to reply.
The "Cold Shoulder": The response felt impersonal or lacked emotion.

Note: Sometimes, these "robotic" signs were actually just because the counselor was busy handling many chats at once or using standard safety scripts, not because they were using AI.

3. What Did People Want?

Most people on the line wanted a human.

21.5% explicitly said, "I don't want to talk to AI; I want a real person."
They felt AI lacked empathy (the ability to truly understand feelings), was untrustworthy, or just unhelpful.
Interestingly, a tiny fraction (0.2%) actually wanted to talk to a bot, but they were the exception.

4. How the Crew Responded (The "Reassurance" vs. "Ignore" Game)

When a passenger asked, "Are you a robot?", the human counselors had two main ways to react:

Strategy A: The "Reassurance" (The most common)
- What they did: They directly said, "I assure you, I am a real human!"
- The Result: This worked well. About 55% of the suspicious clients believed them and felt better. Another 28% just dropped the topic and moved on.
- Analogy: It's like a pilot saying, "I'm a real pilot, not a computer," and the passengers immediately relaxing.
Strategy B: The "Avoidant" (The "Let's Focus on You" approach)
- What they did: They ignored the question and said, "I hear your concern, but let's focus on your problem."
- The Result: This was riskier. About 48% of clients moved on, but 52% got frustrated, kept asking, or even ended the chat.
- Analogy: It's like a pilot ignoring the question and just saying, "Look out the window," which makes the passengers even more nervous.

5. The Big Takeaway

The paper concludes that trust is fragile. Even if an AI is actually helpful, if people think it's AI, they feel less safe and less understood.

The Lesson: When people are in crisis, the feeling of "genuine connection" is everything. If a counselor (human or AI-assisted) sounds robotic, clients will suspect they are talking to a machine.
The Solution: The most effective way to fix this suspicion is honesty and reassurance. When counselors explicitly confirm they are human, the "robot radar" usually turns off, and the conversation can get back to saving the lifeboat.

In short: People are worried they are talking to robots. When they ask, the best thing a counselor can do is simply say, "No, I'm a real person," and mean it. If they try to dodge the question, the client often gets more suspicious.

Technical Summary: "Are you an AI?" Analyzing client suspicion of AI use in crisis counseling

Problem Statement
As artificial intelligence (AI) tools are increasingly deployed in mental healthcare, public trust in these systems remains uncertain. While AI may offer scalability, it is unclear how clients perceive AI involvement in counseling interactions, particularly during crisis moments requiring empathy and connection. There is a critical gap in understanding the frequency of client suspicion regarding AI use, the specific triggers that precipitate these doubts, and how human counselors respond to such challenges. Without this data, designing AI systems that preserve the therapeutic relationship is difficult.

Methodology
The study analyzed a dataset of 75,777 WhatsApp conversations (comprising 2,225,107 messages) collected from the Vandrevala Foundation Emergency Helpline in India between May 31, 2024, and March 31, 2025. All interactions were conducted by trained human counselors without AI assistance.

To identify and characterize instances of AI suspicion, the authors employed a Large Language Model (LLM, GPT-4.1) to process the corpus. The LLM was tasked with:

Detection: Identifying conversations where clients explicitly suspected they were speaking to an AI.
Annotation: Extracting specific variables including client demographics, presenting concerns, attitudes toward AI, triggers for suspicion (e.g., response delays, repetitive phrasing), timing of suspicion within the conversation, counselor response strategies, and subsequent client reactions.

The annotation process was validated through a rigorous inter-rater reliability study. Five human raters annotated 100 conversations, achieving substantial agreement (Cohen's $\kappa \ge 0.61$ ) on single-label categories. The LLM was then evaluated against these human annotations, demonstrating comparable or superior agreement metrics (e.g., $\kappa = 0.82$ for Client Attitude, $\kappa = 0.89$ for Counselor Strategy), supporting its use as a scalable annotator.

Key Results

Prevalence and Demographics: The study identified 1,157 conversations (1.5%) where clients suspected AI use. The proportion of such conversations increased from 0.8% in June 2024 to 2.6% in March 2025. Clients expressing suspicion were statistically more likely to be male (38.9% vs. 28.9%) and aged 18–24 (61.3% vs. 53.0%) compared to the general sample.
Client Attitudes and Beliefs: Among the suspicious clients, 21.5% explicitly preferred human interaction, while only 0.2% expressed a preference for AI. The most common negative beliefs about AI were that it lacks empathy (16.0%), is untrustworthy (11.0%), or is unhelpful (10.6%).
Triggers for Suspicion: Suspicion most frequently arose in the first (38.5%) or second (30.3%) quarter of the conversation. Primary triggers included:
- Questions not directly answered (17.9%).
- Impersonal responses (12.5%).
- Repetitive responses (14.6%).
- Longer response times (8.5%).
- Notably, in 46.6% of cases, there was "no clear trigger" identified in the counselor's behavior.
Counselor Responses and Outcomes:
- Reassurance: In 84.3% of cases, counselors explicitly reassured the client they were human (e.g., "I assure you; this is not ai!"). This strategy led to 54.8% of clients becoming less suspicious and 27.6% moving on to discuss their concerns.
- Avoidant/Redirecting: In 15.7% of cases, counselors redirected the conversation to the client's presenting concern without explicitly addressing the suspicion. This resulted in 47.8% of clients moving on, but 52.2% continued to press the issue, ended the conversation, or expressed frustration.

Significance and Claims
The paper claims that as public familiarity with AI grows, suspicion of AI use in digital care settings is likely to increase, posing potential risks to the therapeutic relationship. The authors highlight a paradox: while blinded reviewers may judge AI-assisted messages as more human and empathetic than purely human ones, clients rate messages less favorably when they suspect AI involvement, regardless of whether that suspicion is correct.

The study concludes that transparency does not always build trust. Instead, the findings suggest that counselors who acknowledge client doubts and provide direct reassurance achieve markedly better outcomes than those who avoid the concern. The authors argue that integrating AI assistance into counseling requires a careful balance between transparency and the perceived authenticity of the therapeutic interaction, noting that the "human" element is a critical component of client trust in crisis scenarios.