🤖 AI

NICE: A Theory-Grounded Diagnostic Benchmark for Social Intelligence of LLMs

This paper introduces NICE, a theory-grounded diagnostic benchmark comprising 137 items across 11 dimensions that evaluates the social intelligence of large language models, revealing that while they achieve high aggregate accuracy, they exhibit consistent weaknesses in specific communication facets such as multi-turn interaction, nonverbal cues, and synchrony.

Original authors: Yunjin Qi, Zhaojun Jiang, Xuan Wu, Hanxi Pan, Yixuan Wang, Yanfang Liu, Xiang Ji, Churu Yu, Chunyuan Zheng, Yingze Chen, Jie He, Liuqing Chen, Zaifeng Gao

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yunjin Qi, Zhaojun Jiang, Xuan Wu, Hanxi Pan, Yixuan Wang, Yanfang Liu, Xiang Ji, Churu Yu, Chunyuan Zheng, Yingze Chen, Jie He, Liuqing Chen, Zaifeng Gao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a new assistant to help you navigate the complex world of human relationships. You don't just want to know if they can answer questions correctly; you want to know if they can "read the room," understand unspoken rules, and know when to speak up or stay quiet.

This paper introduces NICE (Norm, Interaction, Cognition, Experience), a new "report card" designed specifically to test how well Large Language Models (LLMs)—the brains behind AI chatbots—handle these social situations.

Here is the breakdown of what the researchers did and what they found, using simple analogies:

1. The Problem: The "Blind" Test

Before NICE, testing AI's social skills was like taking a driving test where you only had to parallel park.

Old Tests: Some tests only checked if the AI knew a specific fact (like "What is a polite greeting?"). Others put the AI in a chaotic, open-ended conversation where it was impossible to tell why it failed.
The Issue: If an AI got a low score, researchers couldn't tell if it was bad at understanding emotions, bad at following rules, or just bad at talking. It was like saying, "You failed the driving test," without knowing if you couldn't see, couldn't steer, or didn't know the traffic laws.

2. The Solution: The "Social X-Ray" (NICE)

The researchers built NICE to be a diagnostic tool, not just a scorecard. Think of it as an X-ray that breaks down "Social Intelligence" into specific parts so doctors (researchers) can see exactly where the bone is broken.

They created a framework based on psychology, organizing social skills into 4 main categories and 11 specific dimensions:

Norms: Knowing the rules of the game (politeness, ethics).
Interaction: How you talk and act with others.
Cognition: Understanding what others are thinking or feeling.
Experience: Learning from past interactions.

Instead of asking the AI to write a long story, NICE gives it a short scenario (like waiting for a crowded elevator) and asks it to rank three possible responses from "Best" to "Worst." This forces the AI to show it understands the boundaries of social behavior, not just the "correct" answer.

3. The Experiment: AI vs. Humans

The researchers tested 5 of the smartest AI models available (like GPT-5.5 and Claude-Opus-4.7) against a group of real humans.

The Surprise: Overall, the AI models actually scored higher than the humans! They were better at remembering facts, following ethical rules, and managing relationships in a theoretical sense.
The Catch: When the researchers looked at the "X-ray" (the specific dimensions), they found a massive, consistent weakness.

4. The Big Discovery: The "Communication Gap"

While the AI was great at the rules (Norms) and the logic (Cognition), it struggled terribly with Communication (D3).

Think of it like a student who memorized the entire textbook on "How to Be a Friend" but fails when actually talking to a friend.

Where AI Failed: The models were specifically bad at:
- Multi-turn communication: Keeping a conversation going naturally over several turns.
- Nonverbal communication: Understanding tone, body language, or implied meaning (even in text).
- Synchrony: Matching the rhythm and flow of a human interaction.

The "Bow" Example:
The paper gives a funny example: In a scenario where someone bows too deeply (180 degrees), humans immediately recognized this as weird and inappropriate. However, the top AI models thought this was actually a good or neutral response. The AI was so focused on being "polite" that it missed the social boundary that said, "This is too much!"

5. The Conclusion

NICE proves that even the smartest AI models have a specific "blind spot." They are excellent at knowing what to say based on rules, but they are often clumsy at how to say it in a way that feels natural and human.

The paper concludes that to make AI truly socially intelligent, we can't just make them smarter overall; we need to specifically train them to fix these communication gaps, just like a coach would focus on a player's weak foot rather than just telling them to "play better."

In short: AI is a brilliant student of social theory, but it's still a bit awkward at the actual party. NICE is the tool that finally told us exactly why.

Technical Summary: NICE – A Theory-Grounded Diagnostic Benchmark for Social Intelligence of LLMs

Problem Statement

As Large Language Models (LLMs) are increasingly deployed in socially intensive contexts (e.g., emotional companionship, customer service), evaluating their social intelligence has become critical for ensuring the quality and safety of human-AI interaction. However, existing benchmarks suffer from two primary limitations:

Lack of Unified Structure: Benchmarks targeting single abilities (e.g., Theory of Mind, emotion understanding) lack a shared framework to map strengths and weaknesses across the broader social intelligence space.
Lack of Diagnostic Granularity: Open-ended, multi-turn scenarios offer high ecological validity but entangle multiple abilities within a single scenario, making it impossible to trace failures to specific capability deficits.

Consequently, current evaluations often reduce models to aggregate scores that indicate how much they get right but fail to diagnose where they fall short, hindering targeted improvement.

Methodology

The authors propose NICE (Norm, Interaction, Cognition, Experience), the first holistic diagnostic benchmark grounded in social theory and constructed through a rigorous psychometric pipeline. The construction involves four stages:

1. Framework Construction

A social intelligence framework was developed through a top-down systematic literature review and a bottom-up Delphi method involving 23 experts.

Theoretical Basis: The framework integrates social information processing theory (Crick & Dodge, 1994) with eight classical theories of human social intelligence and recent AI reviews.
Structure: The final framework organizes social intelligence into 4 categories, 11 dimensions, and 34 fine-grained capability facets.
Weighting: Structural weights for categories and dimensions were estimated using the Analytic Hierarchy Process (AHP) based on pairwise comparisons by 9 experts.

2. Material Collection and Item Construction

Source: Items were not extracted from raw corpora to avoid construct contamination. Instead, the team adapted 18 existing benchmarks and 43 psychological paradigms.
Design: A pool of 137 items was created, where each item aligns with exactly one capability facet.
Task Format: NICE utilizes a closed-form ranking task. For each scenario, models must rank 3–6 candidate responses from "socially optimal" to "boundary-violating." This tests not only the selection of the best response but also the model's sensitivity to behavioral and normative boundaries.

3. Validation

The benchmark underwent a three-round evaluation and revision process involving 12 interdisciplinary evaluators (blinded to model results). Items were assessed on reliability (clarity, gradient stability), validity (construct alignment), and neutrality (bias). Only items scoring ≥3.5 on all criteria were retained.

4. Evaluation Protocol

Models: Five frontier LLMs were evaluated: GPT-5.5, Claude-Opus-4.7, Gemini-3.1-pro-preview, DeepSeek-v4-pro, and Qwen3.6-plus.
Setup: Zero-shot, greedy decoding settings. Each model ran the 137-item benchmark three times to assess stability.
Baseline: A human reference group of 14 native Chinese speakers completed the same tasks.
Metric: Accuracy is defined strictly as matching the full permutation of the gold ranking; partial agreement receives no credit.

Key Contributions

First Systematic Diagnostic Benchmark: NICE is the first benchmark to apply measurement principles throughout the entire pipeline (framework, item development, validation) for artificial social intelligence, organizing it into a structured hierarchy of 4 categories, 11 dimensions, and 34 facets.
Facet-Level Diagnosis: By tying each item to a single predefined capability facet, NICE enables the tracing of model errors to specific weaknesses rather than absorbing them into dimension-level or overall scores.
Psychometric Rigor: Unlike simple task collections, NICE employs a full psychometric pipeline, including expert consensus, AHP weighting, and iterative validation, ensuring that diagnostic results are interpretable and structurally valid.

Results and Analysis

The evaluation of five frontier LLMs against the human baseline yielded the following insights:

Aggregate Performance: LLMs achieved higher aggregate accuracy ( $M=0.751$ ) compared to humans ( $M=0.704$ ), with a statistically significant gap ( $p < .001$ ).
Dimensional Disparities:
- LLM Strengths: Models significantly outperformed humans in Social Responsibility, Self-consistency, Emotional Utilization, Social Perception, and Adaptive Learning.
- LLM Weaknesses: A consistent weakness was identified in Communication (D3), where humans significantly outperformed LLMs (Human $M=0.518$ vs. LLM $M=0.428$ ; gap of 9.0%).
Granular Diagnosis of Communication: The weakness in Communication was localized to three specific facets:
1. Multi-turn communication: LLMs lagged humans by 52.4%.
2. Nonverbal communication: LLMs lagged by 24.8%.
3. Synchrony: LLMs lagged by 10.5%.
Case Study Findings: Analysis revealed that models often fail to penalize "boundary-violating" behaviors (e.g., excessive politeness like bowing 180 degrees) that humans correctly identify as inappropriate. This suggests models may overvalue explicit deference while lacking sensitivity to contextual norms.
Stability: Failures in the Communication dimension were systematic rather than random; when a model failed a Communication item, it consistently failed it across multiple runs.

Significance and Claims

The paper claims that NICE reframes social intelligence evaluation from a focus on aggregate scores to a theory-grounded diagnosis of socially consequential weaknesses.

Diagnostic Utility: The benchmark demonstrates that high aggregate accuracy can mask critical deficits in specific areas, such as communication nuances and boundary sensitivity, which are vital for safe human-AI interaction.
Actionable Insights: By identifying specific capability facets (e.g., multi-turn communication, nonverbal cues) where models consistently underperform, NICE provides a roadmap for targeted model training and alignment.
Limitations: The authors acknowledge that NICE is currently a static, text-based benchmark developed primarily within a Chinese cultural context. While it approximates real-life situations, it cannot fully capture the dynamics of continuous, real-world social interaction. Future work is needed to extend the benchmark to more complex, interactive, and cross-cultural settings.