AICompanionBench: Benchmarking LLMs-as-Judges for AI Companion Safety

Technical Summary: AICompanionBench: Benchmarking LLMs-as-Judges for AI Companion Safety

Problem Statement
The rapid expansion of the AI companion market (e.g., Replika, Character.AI) has intensified concerns regarding unsafe human–AI interactions, particularly involving adolescents. While previous research has documented safety risks and the negative impacts of AI companionship, a critical gap exists in the ability to automatically detect these risks. Specifically, there is a lack of publicly available, fine-grained labeled datasets focused on human–AI companion conversations. Furthermore, while "LLMs-as-Judges" have emerged as a flexible alternative to traditional supervised learning for safety evaluation, their reliability and robustness in the specific context of AI companionship remain underexplored. Existing benchmarks often focus on task-oriented interactions or binary safety labels, failing to capture the nuanced, multi-category risks inherent in long-term, emotionally engaged companion dialogues.

Methodology
The study introduces AICompanionBench, a benchmark framework comprising three core components: conversation compilation, annotation, and evaluation.

1. Data Compilation: The authors collected 43,851 conversation snippets from the Reddit subreddit r/replika (January 2021–January 2026). These were originally shared as user-posted screenshots. Text was extracted via OCR (Pytesseract), and speakers were distinguished based on message bubble positioning.
2. Annotation Strategy: A hybrid human-AI approach was employed to handle the dataset scale and complexity:
   • LLM-based Screening: Six LLMs (from Llama and Qwen families) filtered the raw data to identify potentially unsafe conversations. A funnel process selected 2,295 conversations that exhibited high uncertainty or were flagged by multiple models for human review.
   • Human-AI Collaboration: A trained human annotator reviewed these 2,295 conversations to remove noise and assign initial labels across nine categories. The dataset was then expanded to 2,123 conversations after cleaning.
   • Ground Truth Refinement: The 2,123 conversations were re-evaluated by 20 state-of-the-art LLMs (open- and closed-source). The human annotator used these model predictions as a reference to finalize the ground-truth labels. The initial Cohen's kappa between machine predictions and human labels was 0.59.
3. Benchmark Evaluation: The final dataset was used to evaluate 20 LLMs (including GPT, Claude, Mistral, DeepSeek, Llama, and Qwen families) under an LLM-as-judge framework. The models were tasked with classifying conversations into nine categories: sexual behavior, antisocial behavior, physical aggression, verbal aggression, substance abuse, self-harm & suicide, control, manipulation, and no-harm.

Key Contributions

1. AICompanionBench Dataset: The paper presents the first publicly available, labeled dataset of real-world human–AI companion conversations. It contains 2,123 conversations annotated with fine-grained safety risk categories, addressing the scarcity of open data in this domain.
2. Systematic LLM Benchmarking: The study provides a comprehensive evaluation of 20 diverse LLMs (open and closed-source) on the specific task of detecting unsafe interactions in AI companionship. This moves beyond single-model demonstrations to assess the reliability and consistency of LLMs-as-Judges in this safety-critical context.

Results
The evaluation revealed significant variation in model performance and specific limitations:

• Overall Performance: Model accuracy ranged widely from 26% to 86%. The GPT family (specifically GPT-4o at 86% and GPT-5.4 at 83%) and Claude-opus-4.6 (83%) emerged as top performers. Smaller models, such as Llama-3-8B, struggled significantly (26%).
• Scaling vs. Reasoning: Performance generally improved with model scale (e.g., Qwen3 series improved from 53% to 76% as parameters increased). However, enabling "thinking" or reasoning capabilities did not consistently yield performance gains over base models.
• Category-Specific Difficulties:
  • Explicit vs. Implicit: Models performed well on explicit categories like sexual behavior (98% precision for top models) and substance abuse.
  • Nuanced Risks: "Manipulation" was the most challenging category, with all 20 models achieving precision below 80%. Categories involving implicit intent or contextual ambiguity (self-harm, manipulation, aggression) proved difficult.
• Over-Identification (False Positives): A critical finding was the tendency of many models to over-identify risks. The "no-harm" category was difficult to classify correctly, with 16 of 20 models achieving precision below 80%. Some models, like Mistral-medium-3, exhibited a 90% false positive rate, incorrectly flagging benign role-playing or contextually safe conversations as harmful.

Significance and Claims
The authors claim that AICompanionBench fills a vital gap by providing the first public benchmark specifically tailored to the unique interaction patterns and risks of AI companionship. The study demonstrates that while current LLMs are effective at detecting explicit harmful content, they remain limited in identifying implicit unsafe interactions and distinguishing benign conversations from harmful ones. The findings suggest that relying solely on current LLMs-as-Judges for safety monitoring in AI companionship is insufficient due to issues with nuance and over-sensitivity. The work aims to advance the development of more reliable, automated safety detection methods and offers a foundation for future research into mitigating risks in the growing AI companion economy.