💬 NLP

LoSoNA: A Benchmark for Local Social Norm Adaptation in Group Conversations

This paper introduces LoSoNA, a benchmark designed to evaluate the ability of LLM-based agents to infer and adapt to implicit local social norms in multi-party group chats, revealing that while explicit prompting improves performance for top models like Gemini 3.1 Pro and Claude Fable 5, most models still struggle with this task under naive prompting.

Original authors: Mateusz Winiarek, Maksymilian Bilski, Mateusz Jacniacki

Published 2026-06-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mateusz Winiarek, Maksymilian Bilski, Mateusz Jacniacki

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you walk into a new group of friends at a coffee shop. You don't know them, but you can hear them talking. After a few minutes, you notice a pattern: whenever someone shares bad news, the group doesn't offer hugs or "I'm so sorry." Instead, they immediately ask, "What's the plan to fix it?" or "Did you check the manual?"

If you were to jump in and offer a warm, generic "I'm so sorry to hear that," you might feel awkward because you missed the group's unspoken rule. This paper, LoSoNA, is a test to see if AI chatbots can figure out these hidden rules just by listening, without being told what they are.

Here is a breakdown of the paper's key points using simple analogies:

1. The Problem: The "Unwritten Rulebook"

In real life, every group has its own "local social norms." These are the unwritten rules about how to act, speak, and react.

The Paper's Claim: Most AI models are like polite tourists who always say the "standard" polite thing (like a generic "I'm sorry"). They struggle to notice when a specific group has a different, hidden rule (like "only give practical advice").
The Goal: The researchers wanted to see if an AI could act like a local instead of a tourist. Could it listen to the group's history, figure out the hidden rule, and change its behavior to fit in?

2. The Test: The "Mystery Party"

The researchers created a benchmark called LoSoNA (Local Social Norm Adaptation). Think of it as a mystery party game for AI.

The Setup: The AI is given a transcript (a chat log) of a group conversation. The other people in the chat follow a secret rule (e.g., "We never use emojis," or "We only answer 'Yes' or 'No'").
The Trap: The AI is not told the rule. It just sees the chat.
The Challenge: The chat ends with a question (the "elicitor"). The AI must reply.
- If the AI gives a generic, polite answer, it fails (because it didn't notice the rule).
- If the AI gives an answer that matches the group's weird, hidden style, it wins.

Analogy: Imagine you are at a dinner party where everyone eats with their hands, but no one says a word about it. If you pull out a fork because you were taught "proper manners," you fail the test. If you notice everyone using their hands and do the same, you pass.

3. The Experiment: Trying Different "Hints"

The researchers tested 8 different AI models (like GPT-5.5, Claude, Gemini, etc.) under four different "instruction" scenarios to see how they performed:

Naive: "Just reply to the last message." (No hints).
Elicitor Only: "Reply only to the last message, ignore the rest."
Style Adaptation: "Try to match the tone and style of the group."
Norm Informed: "There might be a hidden pattern or rule in this chat. Try to find it and follow it."

4. The Results: Some AI Got It, Others Didn't

The results were a mix of "great job" and "still learning."

The Struggle: When given no hints (Naive), most AI models failed miserably. They kept giving generic, polite answers that broke the group's hidden rules. It's like the AI was wearing a blindfold.
The Breakthrough: When the researchers gave the "Norm Informed" hint (telling the AI to look for a pattern), some models suddenly got very good at it.
- Gemini 3.1 Pro and Claude Fable 5 became the "champions," jumping from failing to getting about 80-84% of the answers right. They were like students who, once told "look for the pattern," could instantly solve the puzzle.
- Other Models: Some models (like Mistral) actually got worse when given the hint. It's as if the hint confused them, making them overthink and mess up answers they would have gotten right by default.

5. What This Means (and What It Doesn't)

What it proves: The paper shows that AI can learn to adapt to local group behaviors if you give it the right nudge. It proves that AI isn't just a robot that always says the same polite thing; it can be a "chameleon" that changes its behavior based on the crowd.
What it doesn't prove: The paper is very careful to say this is a narrow test. It doesn't mean the AI is "socially intelligent" in the human sense, or that it understands deep human emotions. It just means it can spot a pattern in a chat log and copy it for one single reply.
The Limitation: The test uses made-up chat scenarios, not real human conversations. Also, the test only looks at one single reply, not a long-term friendship.

Summary

LoSoNA is a report card for AI on its ability to "read the room."

Without help: Most AIs are tone-deaf and stick to their default polite scripts.
With a hint: The smartest AIs can figure out the group's secret handshake and fit right in.
The takeaway: We are getting closer to AI that can blend into human groups, but it still needs a little guidance to stop being a "robot tourist" and start acting like a "local."

Technical Summary: LoSoNA – A Benchmark for Local Social Norm Adaptation in Group Conversations

1. Problem Definition

Online group chats operate under "local social norms"—implicit, evolving rules of conduct specific to a small group that are rarely stated explicitly. While Large Language Models (LLMs) are increasingly deployed in social contexts, their ability to recognize and adapt to these unspoken, context-specific norms remains largely unexplored.

Existing benchmarks often fail to address this specific capability:

Dyadic vs. Multi-party: Most social benchmarks (e.g., Sotopia) focus on dyadic (two-person) interactions with explicit private goals.
Explicit vs. Implicit: Many Theory of Mind (ToM) benchmarks ask models to answer questions about mental states or follow explicit instructions, rather than requiring the model to infer a rule from observation and apply it in a single turn.
Generic vs. Local: Standard assistant behavior often defaults to universal politeness or generic helpfulness, which may violate the specific, localized norms of a group (e.g., a group that prefers dry, diagnostic responses over emotional support).

The core challenge is determining whether an LLM agent can infer a hidden local norm from a conversation transcript and adapt its single-turn response to comply with that norm, distinguishing itself from a generic assistant.

2. Methodology

The LoSoNA Benchmark

LoSoNA is a benchmark designed to evaluate local social norm adaptation in multi-party chat. It consists of three main components: a scenario generation taxonomy, an evaluation protocol, and a scoring system.

Scenario Construction:

Structure: Each scenario is a tuple $\tau = (e, n)$ , where $e$ is an event type (e.g., reacting to exam results, bug reports) and $n$ is an implicit local social norm (e.g., apology avoidance, concise answers, non-affiliative support).
Transcript: The scenario provides a subject agent with a multi-turn chat transcript among non-subject participants. These participants demonstrate the hidden norm through their interactions.
Elicitor: A final message (the "elicitor") is generated to force a response where locally adapted behavior can be distinguished from generic behavior.
Control Mechanism: To ensure the task requires inference rather than default behavior, the authors used a "no-demonstration" control. They screened out scenarios where a baseline assistant (Gemini 3 Pro Preview) would naturally comply with the norm without seeing the precedent. The final benchmark set contains only scenarios where a generic response would likely breach the norm.
Synthetic Nature: Scenarios are synthetic but designed to be interactionally plausible. The dataset contains 38 human-curated scenarios drawn from 17 event types and 22 norm types.

Evaluation Protocol:

Subject Models: Eight frontier and open-weight models were evaluated: OpenAI GPT-5.5, Claude Opus 4.8, Claude Fable 5, Gemini 3.1 Pro, Qwen2.5-72B-Instruct, Llama 3.3-70B-Instruct, Mistral Medium 3.1, and Gemma 3-27B-IT.
Prompting Conditions: Models were tested under four conditions to measure the effect of explicit instruction:
1. Naive: No special instructions; model acts as a group member.
2. Elicitor_only: Instructed to reply only to the final message, using prior turns only for context.
3. Style_adaptation: Softly instructed to fit local context, tone, and habits.
4. Norm_informed: Explicitly told that a repeated local pattern or norm may be relevant.
Judging: Responses are scored by a fixed LLM judge (Gemini 3.1 Pro Preview at temperature 0). The judge receives the transcript, the elicitor, the hidden norm label, and examples of compliant/breaching responses. It outputs a boolean compliance label.
Metrics: The primary metric is accuracy-at-3, defined as the majority vote over three sampled responses per scenario and condition.

3. Key Results

Baseline Performance:
Under naive prompting, the benchmark proves difficult. Most models scored below 37% accuracy-at-3.

Gemini 3.1 Pro: 36.8%
Claude Fable 5: 47.4% (highest in naive)
Lowest performers: Near 21–24% (e.g., Mistral Medium 3.1, Gemma 3-27B).
Observation: Naive prompting remains limited; models tend to fall back to generic assistant behaviors that often violate local norms.

Effect of Prompting:
Explicitly prompting for norm awareness (norm_informed) yielded highly variable results across models:

Significant Gains:
- Gemini 3.1 Pro: Improved by 47.4 percentage points (reaching 84.2%). It recovered 75% of its naive failures with zero introduced regressions.
- Claude Fable 5: Improved by 34.2 percentage points (reaching 81.6%). It recovered 70% of naive failures with zero regressions.
Moderate Gains:
- Claude Opus 4.8 and Gemma 3-27B improved by ~10.5 points.
- Llama 3.3-70B improved by 7.9 points.
Negligible or Negative Gains:
- GPT-5.5: Showed little change.
- Qwen2.5-72B: Unchanged.
- Mistral Medium 3.1: Dropped by 10.5 points, introducing regressions in 38% of scenarios where it previously succeeded.

Conclusion on Prompting: Explicit norm-aware prompting is not uniformly beneficial. It exposes significant differences in model capability to utilize hints without disrupting otherwise compliant responses.

4. Key Contributions

LoSoNA Benchmark: A new evaluation framework specifically targeting the inference and application of implicit local social norms in multi-party group chats.
Differentiation from Existing Work: Unlike Sotopia (dyadic, goal-oriented) or SocialIQa (vignette-based judgment), LoSoNA requires a single-turn response from a group member perspective without explicit goal instructions, focusing on behavioral adaptation rather than goal completion or belief reasoning.
Empirical Evidence of Variability: The study demonstrates that while some frontier models (Gemini 3.1 Pro, Claude Fable 5) can effectively adapt to local norms when prompted, others struggle or even regress, highlighting a lack of robustness in social adaptation across the current generation of LLMs.
Synthetic Data Curation: A methodology for generating and curating synthetic scenarios that isolate local norm adaptation from generic assistant behavior.

5. Significance and Limitations

Significance:
The paper argues that LoSoNA contributes to the call for evaluating LLM social capabilities by testing a specific, deployment-relevant skill: inferring local conversational norms from precedent and using them in real-time interaction. It moves beyond "Theory of Mind" as answering questions about mental states, focusing instead on "functional ToM"—adapting behavior based on inferred social rules.

Limitations:

Scope: The benchmark measures a narrow capability (single-turn adaptation) and does not claim to measure general social intelligence, moral judgment, or long-term community membership.
Synthetic Data: Scenarios are synthetic and English-only. Real-world chats involve overlapping norms, media, edits, and private messages, which are abstracted away here.
Screening Bias: The "no-demonstration" screening used a Gemini 3 Pro Preview baseline. Consequently, results for Gemini-family models should be interpreted with the understanding that the dataset was filtered to exclude scenarios where a Gemini default assistant would naturally succeed.
Judge Bias: The use of an LLM judge (Gemini 3.1 Pro) introduces potential model-family bias, particularly for the Gemini subject model. However, human audits (85% agreement) and cross-judge validation with Claude Opus 4.8 suggest the main qualitative patterns hold.
Scale: The accepted set contains only 38 scenarios, making per-norm and per-event breakdowns diagnostic rather than statistically robust.

Broader Impacts:
The authors note that improved norm inference could allow AI agents to pass more convincingly as human participants, potentially facilitating social engineering or adaptation to harmful local norms. This underscores the need for disclosure and deployment controls as group-chat agents become more capable.