Technical Summary: FriendBench

Problem Statement

Social intelligence requires reasoning about interpersonal relationships based on behavioral cues rather than explicit verbal content. While AI systems are increasingly tasked with mediating human interaction, their ability to infer social dynamics from multimodal behavior remains under-evaluated. Specifically, there is a lack of benchmarks that isolate the inference of prior familiarity (whether two people have met before) from the semantic content of their conversation.

Existing work often relies on scripted dialogue, scene context, or appearance to infer relationships, or targets subjective constructs like emotions and intentions which lack unambiguous ground truth. The authors posit that a "clean" probe of social perception must rely on thin-slice behavioral cues (Ambady and Rosenthal, 1992) where the conversational topic is controlled, forcing the observer to rely solely on how people interact (coordination, response, orientation) rather than what they say.

Methodology

Benchmark Construction (FriendBench)

The authors introduce FriendBench, a binary classification benchmark (Familiar vs. Strangers) derived from the Seamless Interaction dataset (Agrawal et al., 2025).

• Stimuli: 96 balanced dyads (48 familiar, 48 strangers) drawn from naturalistic "Either-Or" ice-breaker interactions.
• Control: Every dyad responds to the same type of forced-choice hypothetical prompt (e.g., "Would you rather fly or be invisible?"). This ensures the conversational topic is matched across classes, preventing semantic content from revealing the answer.
• Modality: Each interaction is evaluated across three conditions:
  1. Text: Transcripts only.
  2. Audio: Speech only.
  3. Video: Audiovisual (speech + visible behavior).
• Design: The dataset is participant-disjoint (no individual appears in more than one dyad) and stratified by recording site and gender composition to rule out confounds. Clips are 20 seconds long, snapped to turn boundaries.

Evaluation Protocol

• Human Baseline: A crowdsourced study with ~90 raters per modality (total ~280 raters). Raters performed a forced-choice task. The design used a planned-missing, 6-block incomplete structure to allow for the modeling of individual rater accuracy and response bias via crossed random effects.
• Model Evaluation: 26 models from seven companies (OpenAI, Google, Anthropic, Alibaba, Mistral, Thinking Machines, Meta) were evaluated in a zero-shot setting. Models received prompts adapted directly from human instructions to ensure task framing parity.
• Metrics:
  • Accuracy: Overall correct classification.
  • Per-class Recall: Sensitivity to "Familiar" vs. "Stranger."
  • Signal Detection Theory: $d'$ (discrimination ability) and criterion $c$ (response bias). A positive $c$ indicates a bias toward "stranger."

Key Contributions

1. FriendBench Benchmark: A multimodal dataset and evaluation framework for inferring dyadic familiarity where the semantic topic is controlled, isolating behavioral cues.
2. Matched Human-Model Dataset: Release of human ratings for all 96 dyads across text, audio, and video, enabling direct comparison of human and model perception under matched conditions.
3. Comprehensive Model Evaluation: Evaluation of 26 state-of-the-art models (proprietary and open-weight) across all three modalities.
4. Analysis of Perception vs. Bias: A methodological contribution distinguishing between discrimination (ability to tell classes apart) and response bias (tendency to favor one class), revealing that models and humans often reach similar accuracy via different mechanisms.

Results

Accuracy and Modality

• Human Performance: Humans perform at chance in text (51.0% crowd accuracy) but improve significantly with richer modalities: 63.5% in audio and 71.9% in video.
• Model Performance: The best models achieve accuracy statistically indistinguishable from the human crowd in all modalities (e.g., best model 66.7% in audio/video vs. human 63.5%/71.9%).
• Modality Gains:
  • Text: Neither humans nor models find reliable signal; performance is near chance.
  • Audio: Both humans and models gain reliable discrimination over text.
  • Video (Audiovisual): Humans gain a further reliable increment from visible behavior (audio $\to$ video gain of +11.0%). Models do not. The best models plateau between audio and video (e.g., Gemini Video 66.7% vs. Gemini Audio 66.7%), failing to exploit visual cues that humans utilize.

Response Bias and Discrimination

While aggregate accuracy is similar, the mechanisms differ:

• Humans: Maintain a balanced decision rule across modalities (criterion $c \approx 0$), correctly identifying both familiar and stranger pairs with relatively balanced recall.
• Models: Exhibit a strong response bias toward "stranger."
  • In text, models lean heavily toward "stranger" (e.g., $gpt\_text\_mini$ has 91.7% recall for strangers but only 20.8% for familiar pairs).
  • In video, the best model ($gemini\_video$) achieves high accuracy ($d' \approx 1.12$) primarily by labeling almost all pairs as "strangers" (91.7% stranger recall, 41.7% familiar recall), whereas the human crowd remains balanced ($c = 0.06$).
• Conclusion: Models achieve parity in accuracy not by matching human discrimination, but by adopting an effective prior that assumes strangers are more common, leading to a skewed decision rule.

Individual Differences and Difficulty

• Rater Traits: Individual social intelligence traits (TSIS) are largely uncorrelated with accuracy in audio and video.
• Dyad Difficulty: Difficulty is a property of the interaction itself. Dyads that are hard for humans are generally hard for models, and this difficulty transfers across modalities (e.g., audio-video correlation $r=0.61$). However, humans and models often miss different classes due to their differing decision rules.

Significance and Claims

The paper claims that while current multimodal models have reached human-level accuracy on this specific social perception task, they do not yet replicate human-like perception.

1. The Visual Gap: The most significant divergence is in the utilization of the visual channel. Humans reliably convert visible behavioral cues into accuracy gains over speech alone; current models capture the vocal signal but under-exploit the visual channel.
2. Bias vs. Discrimination: High accuracy in social perception tasks can mask severe class bias. Models may appear competent by simply defaulting to "stranger," a behavior that would fail in real-world deployment where base rates differ or where missing a "familiar" pair has different consequences.
3. Evaluation Standard: The authors argue that evaluating social perception requires more than a single accuracy metric. Metrics like per-class recall, signal detection statistics ($d'$ and $c$), and item-level difficulty are necessary to reveal whether a model is truly "understanding" the interaction or merely exploiting statistical priors.

The benchmark serves as a tracking mechanism to determine if future models can be brought to read relationships in a balanced manner, utilizing visible behavior as humans do, rather than relying on speech alone or biased priors.