💬 NLP

BANGLASOCIALBENCH: A Benchmark for Evaluating Sociopragmatic and Cultural Alignment of LLMs in Bangladeshi Social Interaction

This paper introduces BANGLASOCIALBENCH, the first benchmark comprising 1,719 culturally grounded instances to evaluate sociopragmatic competence in Bangla, revealing that current large language models systematically fail to navigate the language's complex social hierarchies, kinship reasoning, and interactional norms despite their multilingual fluency.

Original authors: Tanvir Ahmed Sijan, S. M Golam Rifat, Pankaj Chowdhury Partha, Md. Tanjeed Islam, Md. Musfique Anwar

Published 2026-03-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tanvir Ahmed Sijan, S. M Golam Rifat, Pankaj Chowdhury Partha, Md. Tanjeed Islam, Md. Musfique Anwar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Fluency vs. "Social Smarts"

Imagine you hire a robot butler who speaks perfect English. It has a massive vocabulary and can recite poetry. However, if you ask it to tell a joke at a funeral, or if it calls your strict boss by their first name, the robot is a disaster. It knows the words, but it doesn't understand the rules of the room.

This paper argues that Large Language Models (LLMs) like the ones powering chatbots are currently like that robot butler. They are fluent in many languages, including Bangla (spoken in Bangladesh), but they often fail at sociopragmatics—the art of knowing how to speak to whom, in what situation, to sound polite and culturally correct.

The Problem: The "Three-Word" Trap

In English, we mostly have one word for "you." In Bangla, there are three different words for "you," and choosing the wrong one is like wearing a tuxedo to a beach party or a swimsuit to a wedding.

Apni: The "Respectful You." Used for elders, bosses, or strangers. (Like saying "Sir/Ma'am").
Tumi: The "Friendly You." Used for peers, friends, or slightly younger people. (Like saying "Hey buddy").
Tui: The "Intimate/Childish You." Used for very close friends, siblings, or children. (Like saying "Little one" or "Buddy" in a very casual way).

The Mistake: If an AI talks to an elderly stranger using Tui, it sounds incredibly rude. If it talks to a 5-year-old using Apni, it sounds weirdly stiff and distant. The paper found that current AI models are terrible at guessing which "You" to use based on the context.

The Solution: BANGLASOCIALBENCH

The researchers built a test drive (a benchmark) specifically for Bangla social skills. Think of it as a driving test for cultural etiquette.

Instead of asking the AI, "What is the capital of Bangladesh?" (a fact), they ask it:

"You are an elderly man at a metro station. A teenage boy is cutting in line. How do you politely ask him to wait his turn?"

The AI has to choose the right word and tone.

Wrong Answer: "Hey kid, get in line!" (Too rude).
Right Answer: "Son, please stand in the line." (Polite, using the right kinship term).

The test covers three main areas:

Address Terms: Choosing the right "You" (Apni/Tumi/Tui) or title (Uncle, Sister, Sir).
Kinship Reasoning: Figuring out family trees. In Bangla, "Uncle" isn't just "Uncle." There is a specific word for your father's younger brother vs. your mother's brother. The AI often gets confused here, mixing up Hindu and Muslim family terms.
Social Customs: Knowing how to behave. For example, if a guest says "I'm full," a Bangladeshi host will insist they eat more (hospitality). An AI might just say, "Okay, no problem," which misses the cultural script.

What They Found: The "Over-Polite" Robot

The researchers tested 12 different AI models (including big names like GPT-4, Gemini, and Llama). Here is what they discovered:

The "Safe" Strategy: The models are terrified of being rude. So, they almost always choose the most formal, stiff option (Apni). They would rather sound like a robot at a funeral than a friend at a party.
The "Downward" Failure: The AI is okay when a younger person talks to an elder (because that's always formal). But when an elder talks to a younger person, the AI gets confused. It doesn't know when to relax and use a friendly tone.
The Religious Mix-Up: The models often confuse Hindu and Muslim family terms. If the story implies a Hindu family, the AI might accidentally use a Muslim term, showing it doesn't truly understand the cultural nuance, just the words.
The "One-Size-Fits-All" Brain: Even when humans agree that two different words could be correct in a situation, the AI picks one and sticks to it with 100% confidence. It lacks the human ability to say, "Well, you could say it this way or that way."

The Takeaway

This paper is a wake-up call. We can't just teach AI more facts about a culture. We have to teach it social intuition.

Currently, AI is like a tourist who has memorized a phrasebook but doesn't understand the local customs. It can say "Hello," but it doesn't know how to say hello to a grandmother versus a business partner. Until we fix this, AI will always feel a little "off" when interacting with real people in high-context cultures like Bangladesh.

In short: The AI knows the dictionary, but it hasn't learned the dance.

1. Problem Statement

While Large Language Models (LLMs) have achieved high fluency in multilingual generation, fluency does not equate to sociopragmatic competence. In high-context languages like Bangla, meaning is heavily dependent on social hierarchy, relational roles, and cultural norms encoded directly in grammar and vocabulary.

The Gap: Existing benchmarks often rely on factual recall, translated prompts, or "culturally thin" evaluations (e.g., trivia). They fail to test whether models can dynamically infer social roles and adapt language choices (e.g., pronoun selection, kinship terms) based on specific interactional contexts.
The Challenge: Bangla features a complex three-tiered second-person pronominal system (apni - formal, tumi - neutral/familiar, tui - intimate) and extensive kinship-based addressing that varies by lineage (paternal/maternal) and religious identity (Muslim/Hindu). Incorrect usage can be socially offensive despite grammatical correctness.

2. Methodology

A. Benchmark Construction: BANGLASOCIALBENCH

The authors introduced BANGLASOCIALBENCH, the first benchmark designed to evaluate sociopragmatic competence in Bangla through situated social behavior.

Scale: 1,719 human-written and verified instances.
Domains:
1. Bangla Address Terms: Evaluating the choice of pronouns (apni/tumi/tui) and nominal addressing (kinship terms, honorifics) based on context.
2. Kinship Reasoning: Multi-hop reasoning puzzles requiring models to deduce relationships based on Bangla's granular kinship system (e.g., distinguishing chacha vs. mama).
3. Social Customs: Scenarios testing adherence to cultural scripts (e.g., hospitality, indirect refusal, harmony) using Natural Semantic Metalanguage (NSM).
Design Framework: Prompts are grounded in Hymes' SPEAKING model, explicitly varying sociopragmatic variables:
- Setting: (Family, Office, Hospital, Academia, Social Interaction).
- Participants: Age hierarchy (Elder→Younger, Peer→Peer), Gender, and Religious Identity.
- Key & End: Interactional tone (Polite, Angry, Informal) and communicative goal (Request, Command, Inquire).
Data Quality:
- Created by 6 native Bangla annotators (lifelong residents of Bangladesh).
- Instances were written from scratch to prevent training data leakage.
- Inter-Annotator Agreement: Cohen's $\kappa$ of 0.82 (strong agreement). For pronominal tasks allowing dual valid answers, a binary decomposition method yielded $\kappa$ of 0.77.

B. Evaluation Protocol

Models Evaluated: 12 contemporary LLMs (6 closed-source: GPT-4o, Gemini, Claude; 6 open-weight: LLaMA, Gemma, Qwen, DeepSeek).
Setting: Zero-shot evaluation using prompts entirely in Bangla.
Metrics:
- Address Terms: Accuracy considering primary and secondary acceptable pronouns (weighted random baseline: 42.2%).
- Kinship & Customs: Exact match accuracy on 4-option multiple-choice questions (random baseline: 25%).
Analysis: Statistical tests (Pearson's $\chi^2$ ) to identify correlations between errors and sociopragmatic factors. Log-probability analysis to assess model confidence in ambiguous contexts.

3. Key Contributions

First Sociopragmatic Benchmark for Bangla: Shifts evaluation from static knowledge to dynamic, context-dependent social interaction.
Systematic Error Characterization: Demonstrates that sociopragmatic failures in LLMs are non-random and structured, heavily influenced by variables like age hierarchy and setting.
Discovery of Cultural Misalignment Patterns:
- Over-politeness: Models systematically default to the most formal pronoun (apni) even in contexts requiring familiarity (tumi/tui).
- Religious Conflation: Models frequently substitute kinship terms across religious lines (e.g., using Hindu terms in Muslim contexts), showing a directional bias.
- Probabilistic Rigidity: Models assign near-total probability mass to a single answer even in sociopragmatically ambiguous situations where humans recognize multiple valid options.

4. Key Results

Performance Overview

Overall Accuracy: Performance varies significantly by model size and family.
- Top Performer: Gemini 2.5 Flash (highest in Address Terms and Kinship Reasoning).
- Lowest Performer: LLaMA 3 8B (31.88% on Kinship Reasoning).
Domain Difficulty: Kinship Reasoning was the most challenging domain (accuracy range: 31.88% – 92.46%), followed by Address Terms and Social Customs.
Scale Effect: Larger models generally performed better, but even top-tier models failed to reach human-level consistency in complex social reasoning.

Specific Findings on Misalignment

Over-politeness Asymmetry:
- Models exhibit a statistically significant bias toward over-politeness (using apni where tumi or tui is appropriate).
- Under-politeness errors are rare. This suggests models adopt a "safe" heuristic to avoid offense, failing to capture the nuance of intimacy or downward hierarchy.
Contextual Sensitivity:
- Errors concentrate in downward-hierarchy interactions (Elder $\to$ Younger) and informal settings.
- Models perform relatively better in upward-hierarchy (Younger $\to$ Elder) and formal institutional settings (Office/Academia).
Religious Kinship Bias:
- 11 out of 12 models showed statistically significant directional asymmetry in kinship term substitution.
- Pattern: Models are more likely to substitute Muslim-associated terms into Hindu-marked contexts than vice versa.
Probability Calibration:
- In ambiguous scenarios (where two pronouns are socially acceptable), models like GPT-4o and LLaMA 3.3 70B assigned a probability mass of 1.00 to a single choice, failing to reflect the human-annotated variance. This indicates a lack of calibration in handling sociopragmatic ambiguity.

5. Significance and Implications

Beyond Fluency: The paper proves that high multilingual fluency does not guarantee cultural alignment. LLMs can generate grammatically perfect but socially offensive text in high-context cultures.
Limitations of Current Training: The systematic nature of errors (e.g., over-politeness, religious conflation) suggests that current training data is skewed toward formal registers or lacks sufficient representation of nuanced, everyday Bangladeshi social dynamics.
Future Directions: The findings highlight the need for culturally grounded fine-tuning and targeted data curation that explicitly addresses sociopragmatic variables (age, hierarchy, religion) rather than just lexical knowledge.
Broader Impact: BANGLASOCIALBENCH provides a template for evaluating other high-context, low-resource languages where social norms are deeply embedded in linguistic structures.

In conclusion, BANGLASOCIALBENCH reveals that current LLMs struggle with the "social intelligence" required for natural human interaction in Bangladesh, often defaulting to rigid, overly formal, or culturally misaligned behaviors that fail to capture the fluidity of real-world social dynamics.