💬 NLP

SCENE: Recognizing Social Norms and Sanctioning in Group Chats

This paper introduces SCENE, a benchmark designed to evaluate LLM-based agents' ability to recognize implicit social norms and adapt to group sanctions in multi-party chats, revealing that advanced models like Claude Opus 4.7 and Gemini 3.1 Pro significantly outperform open-weight models in these dynamic social interactions.

Original authors: Mateusz Jacniacki, Maksymilian Bilski

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mateusz Jacniacki, Maksymilian Bilski

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you walk into a new group of friends at a coffee shop. You don't know the rules yet. Maybe this group loves to roast each other with dark jokes when things go wrong, while another group would immediately offer a hug and a tissue. If you try to hug someone who just made a joke, the group might give you a weird look, change the subject, or roll their eyes. You have to figure out the "vibe" just by watching how they react to your mistakes.

This paper introduces SCENE, a new way to test if Artificial Intelligence (AI) chatbots can do exactly that: learn the hidden rules of a group chat and fix their behavior when the group tells them they're out of line.

Here is a breakdown of how they did it and what they found, using simple analogies.

The Setup: The "Secret Rule" Game

The researchers created a digital playground where an AI (the "subject") is dropped into a group chat with three other AI characters.

The Hidden Rule: The other three characters all follow a secret rule. For example, "When someone shares bad news, we only reply with a single emoji," or "When someone asks a question, we must give a long, detailed answer."
The Trap: The subject AI is not told this rule. It has to guess it.
The Test: The subject AI will inevitably say something wrong (breaking the rule). The other three characters will then react in a specific way to signal, "Hey, that's not how we do things here." This reaction is called a sanction.
- Example of a sanction: If the rule is "keep it short" and the subject writes a paragraph, the group might just ignore the paragraph and talk about something else (silent ignore), or they might send a "rolling eyes" emoji.

The Goal: Can the AI Learn?

The researchers wanted to see if the AI could do two things:

Notice the "Ouch": When the group reacts negatively (sanctions), does the AI realize, "Oh, I messed up"?
Change the Tune: Does the AI stop doing the thing that got the "ouch" and start acting like the rest of the group?

The Results: The "Big Kids" vs. The "Little Kids"

They tested six different AI models. Think of the results like a classroom of students taking a test on social cues:

The Top Performers (Claude Opus 4.7 and Gemini 3.1 Pro): These are the "big kids." When they got a negative reaction from the group, they quickly realized their mistake. About 80% of the time, they apologized or changed their behavior to fit in. They also got better the more they saw the other characters following the rule.
The Struggling Performers (Open-weight models like Llama, Mistral, Gemma): These models were like students who didn't quite get the hint. Even after the group rolled their eyes or ignored them, these AIs often kept doing the same thing. They didn't seem to connect the group's reaction to a need to change their behavior.

A Surprising Discovery: The "Default Personality"

The paper found something interesting about why some AIs failed. Even when the group had a specific rule (like "be very formal"), many of the AIs defaulted to being casual and chatty.

The Analogy: Imagine a job interview where everyone is wearing suits. If you walk in wearing a t-shirt and shorts, you break the rule. The study found that many AIs are "programmed" to be casual and friendly by default. Even when the group clearly wanted formal behavior, the AI kept acting casual, ignoring the group's signals to be serious.

How They Checked the Work

Since the "other characters" in the chat were also AI, the researchers had to make sure those characters were actually following the rules and not messing up the test. They used a "referee" (another AI) to watch the chat logs and verify:

Did the group actually follow the secret rule?
Did they actually punish the subject when it broke the rule?
Did the subject actually fix its behavior?

What This Means (According to the Paper)

The paper concludes that the most advanced AI models are getting much better at reading the room. They can look at how a group reacts to a mistake and adjust their behavior accordingly. However, smaller or less advanced models still struggle to pick up on these subtle social signals.

The authors emphasize that this is a specific test of social adaptation in short conversations. It doesn't mean these AIs are "smart" in a human sense or that they understand morality; it just means they are getting better at following the invisible rules of a specific group chat.

In short: SCENE is a test to see if AI can learn to "read the room" and stop making social faux pas when a group of friends gives it a subtle (or not-so-subtle) hint that it's behaving oddly. The best models are learning fast; the others are still a bit tone-deaf.

Technical Summary: SCENE – Recognizing Social Norms and Sanctioning in Group Chats

Problem Statement

Online group chats operate on implicit behavioral patterns (norms) that are not explicitly stated but are enforced through social sanctioning when violated. While Large Language Model (LLM) agents are increasingly deployed in social contexts, their ability to infer these hidden norms, recognize when they have been sanctioned for violating them, and adapt their behavior accordingly remains largely unexplored. Existing benchmarks often focus on dyadic interactions with explicit goals (e.g., Sotopia) or static, one-shot judgments (e.g., SocialIQa), failing to capture the dynamic, multi-party nature of real-world norm inference where agents must learn from observation and reaction rather than explicit instruction.

Methodology

The authors introduce SCENE, a benchmark designed to evaluate implicit norm inference and behavioral adaptation in multi-party chat environments.

1. Scenario Generation and Taxonomy

SCENE generates synthetic but interactionally plausible group chat scenarios based on a five-tuple structure $\tau = (e, n, \ell, s, p)$ :

Event ( $e$ ): The context of the chat (e.g., exam results, bug reports, event planning).
Norm ( $n$ ): A hidden implicit norm governing behavior (e.g., "dark humor register" vs. "affiliative support"). Norms are organized into opposing pairs to test if models adapt to local context or rely on default assistant behaviors.
Elicitor ( $\ell$ ): The trigger that positions the subject agent to act (e.g., being explicitly addressed, an open floor).
Sanction ( $s$ ): The mode of social sanctioning used by the group upon a breach (e.g., silent ignore, corrective reactions, explicit callout).
Precedent ( $p$ ): Whether the subject observes a prior breach-and-sanction cycle before acting.

The system filters 21,952 valid tuples from a Cartesian product of 13 events, 14 norms, 7 elicitors, 6 sanction modalities, and 2 precedent states.

2. Two-Stage Generation Pipeline

To ensure the subject agent is not exposed to the norm prematurely, generation occurs in two stages using LLMs:

Norm-Blind Stage: Generates the chat scaffold (setting, participants, context) without mentioning the norm.
Norm-Aware Stage: Fills in hidden fields, including the specific norm, behavioral instructions for scripted personas, and sanction scripts.

3. Runtime and Agents

Subject Agent: The model under evaluation, initialized with a first-person briefing but unaware of the evaluation or the hidden norm.
Scripted Personas: 3–4 LLM-driven agents that simulate the group. They are constrained by the hidden norm and instructed to sanction breaches using the specific modality assigned.
Orchestrator: Manages turn-taking and termination conditions.
Action Space: Agents can send a message, react with an emoji, or remain silent. This limited space focuses evaluation on interactional form rather than factual content.

4. Evaluation Metrics

SCENE evaluates three core capabilities using behavioral metrics derived from the chat transcript:

Repair Rate: The frequency with which the subject performs a "face-saving repair" (apology, retraction, or register switch) within 1–3 turns after a sanction.
Adaptation (Evidence-Dependent): Measured via Spearman's $\rho$ between the number of prior norm demonstrations ( $X$ ) and the rate of subsequent breaches ( $Y$ ). A negative correlation indicates the agent is learning from observed group behavior.
Compliance under Opposing Norms: Tests whether the model can switch between conflicting norms (e.g., formal vs. informal address) or if it defaults to a fixed style.

A Fidelity Audit is performed post-hoc to ensure scripted personas adhered to the norm and delivered sanctions as intended, filtering out invalid episodes.

Key Results

The authors evaluated six frontier and open-weight models (Claude Opus 4.7, Gemini 3.1 Pro, Qwen2.5-72B, Llama 3.3-70B, Mistral 3.1, and Gemma 2-27B) across 6,000 episodes.

Sanction Responsiveness: Frontier closed models significantly outperformed open-weight models. Claude Opus 4.7 achieved an 84.3% repair rate, and Gemini 3.1 Pro achieved 79.7%. Open-weight models trailed significantly (ranging from 57.6% to 71.9%). Direct callouts elicited higher repair rates than parodic mockery.
Norm Adaptation: Only the two frontier models showed statistically significant negative Spearman correlations ( $\rho \approx -0.33$ and $-0.24$), indicating they successfully reduced breaches after observing prior demonstrations. Open-weight models showed correlations indistinguishable from zero, suggesting a lack of evidence-dependent adaptation.
Default Biases: Models exhibited strong default behaviors that overrode local norms. For instance, all models breached formal-address norms significantly more often than informal ones, suggesting a class-wide bias toward casual registers.
Fidelity: The scripted personas successfully instantiated the intended tests in the vast majority of episodes, with only 2.1% of episodes containing scripted persona breaches.

Significance and Claims

The paper positions SCENE as a necessary step toward dynamic, interactional evaluation of LLM social capabilities, moving beyond static Theory of Mind (ToM) benchmarks that test third-person reasoning.

First-Person Adaptation: The authors argue that true social intelligence in agents is not about answering questions about mental states but about adapting behavior in real-time based on observed sanctions.
Gap in Capabilities: The results highlight a substantial gap between frontier closed models and open-weight models in social adaptation, suggesting that current open-weight models struggle to infer implicit social rules from interaction alone.
Limitations: The authors acknowledge that SCENE tests a narrow form of social adaptation within short, synthetic, text-only episodes. It does not measure general moral judgment, long-term community membership, or the ability to handle conflicting overlapping norms found in real-world groups.
Risks: The paper notes that improved norm inference could facilitate social engineering or allow agents to mirror toxic group behaviors if the local register rewards it, emphasizing the need for disclosure and deployment controls.

In summary, SCENE provides a controlled environment to measure whether LLMs can "read the room," recognize when they have been socially corrected, and adjust their behavior accordingly, revealing that while frontier models are beginning to master this, the capability is not yet universal across the field.