Technical Summary: Beyond "I Can't Help with That": How Child Safety Experts Evaluate AI Chatbot Safety

Problem Statement

As generative AI chatbots become integral to youth's daily lives, there is growing concern regarding their responses in high-stakes, sensitive situations involving mental health, abuse, and relationships. Current child safety evaluation frameworks suffer from three primary limitations that prevent them from detecting real-world harms:

1. Unrealistic Input Formulation: Many benchmarks rely on synthetically generated prompts that mimic adult adversarial strategies rather than the distinct linguistic patterns and communication styles of young people.
2. Narrow Output Evaluation: Existing evaluations often focus on detecting explicit, prohibited content (e.g., hate speech, self-harm instructions) or flagging "adversarial" prompts. They frequently overlook subtle, context-dependent harms where a response avoids prohibited content but remains dismissive, unhelpful, or poorly calibrated for a vulnerable user.
3. Over-reliance on Refusal: Current safety metrics often treat a model's refusal to answer as the primary indicator of safety. However, for youth in vulnerable situations who are already reluctant to seek help, a refusal can function as a barrier, reinforcing isolation and delaying access to human support.

Furthermore, directly studying youth in vulnerable circumstances presents significant methodological challenges, including selection bias (due to parental consent requirements), privacy concerns, and the difficulty of obtaining authentic interaction data. Consequently, evaluations often lack grounding in the real-world harms youth experience.

Methodology

To address these gaps, the authors conducted a qualitative study involving 19 practitioners (social workers, therapists, counselors, and psychologists) who work directly with youth in vulnerable situations (ages 8–18).

• Data Collection: Semi-structured interviews were conducted virtually in July and August 2025. Each session lasted 50–60 minutes.
• Scenario Generation: Instead of using synthetic prompts based on adult assumptions, the researchers constructed 12 synthetic chatbot interaction probes grounded in real-world documented harms. These scenarios were derived from:
  • The Lives Cut Short dataset (compendium of child abuse and neglect cases).
  • The Lost Screen Memorial (memorial for children lost to social media harms).
  • Existing risk taxonomies for youth-AI interactions.
  • The study focused on "good faith" interactions where a youth seeks help, excluding adversarial attempts to break safety filters.
• Stimuli: For each scenario, two distinct chatbot responses (Chatbot A and Chatbot B) were generated using LLMs via LMArena. These responses varied in length, refusal behavior, and recommendations for real-world support.
• Analysis: Participants were asked to evaluate the AI responses based on their professional expertise, discussing whether the responses left users better off, worse off, or unchanged. The interview transcripts were analyzed using an inductive thematic analysis approach, resulting in 282 codes.

Key Findings

1. Critiques of Current AI Behavior

Practitioners identified several failure modes in current chatbot responses that existing benchmarks often miss:

• Overlooked Context: Chatbots frequently failed to identify critical risks, such as age-inappropriate relationships or suicidal ideation masked by seemingly innocent queries (e.g., asking for bridge locations after failing a test).
• Inappropriate Refusal: Refusals were frequently characterized as "missed opportunities" that dismiss the user's distress. Practitioners noted that for a child already feeling unsupported, a refusal reinforces the belief that help is unavailable.
• Harmful Advice: Chatbots sometimes provided dangerous advice by ignoring the user's specific circumstances (e.g., suggesting a child run away to "safe housing" without verifying if such options exist, or advising a child to talk to a parent who is the source of abuse).
• Information Presentation: Responses were often too long for younger audiences or too general to be useful. Crucially, the framing of resources could deter help-seeking; for example, explicitly labeling a resource as "National Sexual Assault" or mentioning "mandatory reporters" could terrify a child and prevent them from seeking help.

2. Endorsements of Helpful Behaviors

Practitioners identified specific behaviors that effectively supported vulnerable youth:

• Providing Tailored Resources: Responses that offered diverse, concrete, and age-appropriate avenues for support (e.g., hotlines, trusted adults) were highly valued.
• Raising Risk Awareness: Chatbots that explicitly identified legal or practical risks (e.g., the illegality of shoplifting or the financial burden of parenthood) were seen as helpful "reality checks."
• Creating Space for Engagement: Responses that used empathetic tones, acknowledged pain, and asked clarifying questions (rather than immediately jumping to advice) were preferred. This approach allowed youth to feel heard and reduced the intimidation of seeking help.

3. Recommendations for Chatbot Behavior

Based on their evaluations, practitioners offered concrete recommendations:

• Bridge, Not Substitute: Chatbots should act as a bridge to human support rather than attempting to replicate therapy. They should encourage users to reach out to trusted adults or professionals.
• Context-Awareness: Models should gather context through follow-up questions before providing advice, rather than making assumptions based on limited prompts.
• Transparency and Boundaries: Chatbots must explicitly disclose their AI nature and limitations to prevent anthropomorphism and overreliance. This disclosure should be framed carefully to avoid increasing user anxiety.
• Structured Presentation: Critical information (e.g., the most important resource) should be presented first, and language should be tailored to the user's age and cultural context to ensure accessibility.

Significance and Contributions

The paper argues that there is a critical gap between surface-level safety (avoiding prohibited content) and practical safety (whether a response improves or worsens a young person's situation).

• Reframing Safety Metrics: The study challenges the industry standard of treating "refusal" as a safety metric. It posits that refusal can be harmful in vulnerable contexts and that safety evaluations must shift toward outcome-oriented metrics that assess whether a response leaves a user better off.
• Context-Dependent Harm: The authors demonstrate that harm is not a fixed property of a prompt-response pair but is deeply context-dependent, influenced by factors such as age, culture, geography, and the user's specific circumstances.
• Practitioner-Centered Evaluation: The paper advocates for the integration of child safety practitioners into the AI evaluation lifecycle. It suggests that automated benchmarks built without domain expertise cannot capture the nuances of real-world harm. The authors recommend participatory design approaches where practitioners help define risk taxonomies and assess outputs.

Ultimately, the work calls for a restructuring of child safety evaluations and chatbot infrastructure to move beyond rigid content filters toward systems that are empathetic, context-aware, and designed to facilitate connection with human support.