💬 NLP

ADAB: Arabic Dataset for Automated Politeness Benchmarking -- A Large-Scale Resource for Computational Sociopragmatics

This paper introduces ADAB, a large-scale annotated Arabic dataset comprising 10,000 samples from diverse online platforms and dialects, designed to advance computational sociopragmatics by providing a benchmark for politeness detection across polite, impolite, and neutral categories.

Original authors: Hend Al-Khalifa, Nadia Ghezaiel, Maria Bounnit, Hend Hamed Alhazmi, Noof Abdullah Alfear, Reem Fahad Alqifari, Ameera Masoud Almasoud, Sharefah Al-Ghamdi

Published 2026-02-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hend Al-Khalifa, Nadia Ghezaiel, Maria Bounnit, Hend Hamed Alhazmi, Noof Abdullah Alfear, Reem Fahad Alqifari, Ameera Masoud Almasoud, Sharefah Al-Ghamdi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking into a bustling, noisy marketplace in the Middle East. In this market, people speak in many different accents (dialects), use complex metaphors, and rely heavily on unspoken social rules to show respect or disrespect. Sometimes, a phrase that sounds like a blessing is actually a curse, depending on the tone and who is speaking.

Now, imagine you are trying to build a robot (an AI) to walk into this market and understand who is being polite and who is being rude. That is exactly the challenge this paper tackles.

Here is the story of ADAB, the new tool created to teach computers how to understand Arabic politeness.

1. The Problem: The "Lost in Translation" Robot

For a long time, computers have been great at understanding English politeness. If someone says "Please" or "Thank you," the computer knows they are being nice. But Arabic is different. It's like trying to teach a robot to understand a dance where the steps change depending on whether you are in Egypt, Saudi Arabia, or Morocco.

The Diglossia Puzzle: Arabic has two "faces": a formal version (like a suit and tie) and many dialect versions (like casual street clothes). A robot trained only on the formal version gets confused when people speak casually.
The Hidden Meaning: In Arabic, you can be incredibly polite by using a religious phrase, or incredibly rude by using the same phrase with a sarcastic tone. A standard computer sees the words but misses the "vibe."

Until now, there was no big library of examples to teach the robot these nuances. It was like trying to teach someone to swim without ever giving them a pool to practice in.

2. The Solution: Building the "ADAB" Pool

The researchers built a massive new dataset called ADAB (which means "Politeness" in Arabic). Think of this as a giant, organized library containing 10,000 real-life conversations scraped from the internet.

Where did they get the texts? They didn't just write fake sentences. They collected real comments from:
- YouTube: People reacting to videos (the "street" talk).
- Online Shopping: People reviewing clothes (the "customer service" talk).
- Twitter: People arguing or chatting about news (the "public square" talk).
- Banking Apps: People complaining about money (the "formal complaint" talk).
The Human Touch: To label these texts, they didn't just use a computer. They hired two experts with PhDs in Arabic language. These experts acted like detectives, reading every sentence to decide:
1. Polite: "You are a blessing," "May God protect you."
2. Impolite: "You are foolish," "Go away."
3. Neutral: "The product is okay," "I bought this."

They also tagged why something was polite or rude (e.g., "This is a prayer," "This is an insult," "This is sarcasm").

3. The Big Test: Who is the Best Detective?

Once they built the library, they put 40 different types of "robots" (AI models) to the test to see which one could best read the room. They tested three generations of robots:

The Old School Robots (Traditional Machine Learning): These are like calculators. They look for specific keywords (like "thank you") but don't understand the whole sentence.
- Result: They were okay, but often missed the subtle jokes or sarcasm.
The Super-Readers (Transformers/LLMs): These are the modern AI giants (like the ones behind ChatGPT). They read the whole sentence and understand context.
- Result: They did much better.
The "Local" vs. "Global" Robots: They tested robots trained on all languages versus robots trained specifically on Arabic.
- The Winner: The MARBERT robot (an Arabic-specialist) won the race. It scored a 91% accuracy.
- The Lesson: Just like a tourist might struggle with local slang, a "global" AI struggled with Arabic nuances. The "local" AI, which had studied Arabic dialects and social media slang, understood the culture much better.

4. Where the Robots Still Stumble (The Glitches)

Even the best robot made mistakes. The researchers found four main reasons why:

The "Safe Zone" Bias: The robots were too scared to call anything "rude" or "super polite." They defaulted to saying everything was "Neutral" (just okay). It's like a robot that thinks, "I'm not 100% sure, so I'll just say it's fine."
The Religious Trap: Some phrases use religious words that can be a blessing or a curse depending on the context. The robots often got confused here.
The Slang Wall: The robots struggled with specific dialects (like Egyptian or Maghrebi Arabic) because they hadn't seen enough examples of them.
The "Subtle Insult": Sometimes people are rude without using bad words (passive aggression). The robots missed these "soft" insults.

5. Why Does This Matter?

Why do we care if a robot knows the difference between "God bless you" and "God help you"?

Customer Service: Imagine a chatbot that knows when a customer is getting angry and switches to a super-polite tone to calm them down.
Social Media: Imagine a tool that can spot bullying or harassment in Arabic comments before it hurts someone, even if the bully is using clever, hidden insults.
Respect: It helps technology respect the culture it is serving, rather than treating all languages as if they were English.

The Bottom Line

This paper is like building the first dictionary of feelings specifically for the Arabic-speaking world. It shows us that while AI is getting smarter, it still needs to learn the local culture, the dialects, and the hidden meanings of politeness to truly understand us. The ADAB dataset is the textbook that will help future robots learn these lessons.

1. Problem Statement

Despite the critical role of politeness in human communication, computational resources for Arabic politeness detection are severely under-explored compared to English. Existing research faces several specific challenges:

Linguistic Complexity: Arabic exhibits diglossia (coexistence of Modern Standard Arabic and numerous dialects), rich morphological structures where politeness markers are embedded in verb conjugations and pronouns, and right-to-left script nuances.
Cultural Nuance: Arabic politeness is deeply rooted in Islamic values, social hierarchies, and regional variations (Gulf, Egyptian, Levantine, Maghrebi), making simple translation-based approaches ineffective.
Resource Gap: There is a lack of large-scale, annotated datasets that capture these sociopragmatic phenomena across diverse domains (social media, e-commerce, customer service).

2. Methodology: The ADAB Dataset Construction

The authors introduced ADAB (أدب), a large-scale annotated Arabic dataset designed to bridge this gap.

Data Collection:
- Volume: 10,000 Arabic texts.
- Sources: Four diverse domains: YouTube comments (~~70k raw), Product Reviews (Shein, ~6k), Twitter/X posts (ArSAS, ~20k), and Mobile Banking App reviews (~~67k).
- Diversity: Includes Modern Standard Arabic (MSA) and four major dialects (Gulf, Egyptian, Levantine, Maghrebi).
- Preprocessing: Removal of political content, profanity, religious extremism, and incoherent utterances to focus on genuine politeness phenomena.
Annotation Framework:
- Theoretical Basis: Synthesizes classical Arabic rhetoric (e.g., al-qawl al-layyin - gentle speech) with contemporary pragmatic theories (Lakoff, Leech).
- Classification Scheme: Three-way classification: Polite, Impolite, and Neutral.
- Fine-Grained Features: Each text is annotated with one of 16 specific categories (e.g., Gratitude, Hospitality, Insult, Sarcasm, Prayers). Notably, the category of "Prayers" captures the unique Arabic distinction between blessings (Du'ā' Lak) and curses (Du'ā' 'Alayk) which share linguistic forms.
- Annotators: Two PhD-level Arabic linguists with expertise in multiple dialects.
- Quality Control: Achieved a Cohen's Kappa ( $\kappa$ ) of 0.703, indicating substantial inter-annotator agreement.
Dataset Statistics:
- Class Distribution: Neutral (70.72%), Polite (19.12%), Impolite (10.16%).
- Average Length: 13.90 words.

3. Experimental Setup & Evaluation

The authors conducted a comprehensive benchmarking study using 40 model configurations across three paradigms:

Traditional Machine Learning (12 models): Logistic Regression, SVM, XGBoost using sparse features (TF-IDF) and dense embeddings (FastText, GloVe, Word2Vec).
Transformer-Based Architectures (10 models):
- Arabic-specific: AraBERTv02, MARBERT, CAMeLBERT-mix.
- Multilingual: XLM-RoBERTa, mDeBERTa, RemBERT, InfoXLM, mBERT.
Large Language Models (18 configurations):
- Proprietary: GPT-4 mini, Claude-sonnet-4.5.
- Open-weights: Fanar-1-9B, ALLaM-7B.
- Multilingual: DeepSeek V3, GPT OSS 20B.
- Settings: Evaluated under Zero-shot and Few-shot (1-shot, 3-shot) scenarios.

Metrics: Accuracy, Micro-F1, and Macro-F1 (to address class imbalance).

4. Key Results

Performance Hierarchy: Transformer-based models significantly outperformed both Traditional ML and LLMs.
- Top Performer: MARBERT achieved the highest scores: Accuracy = 91.19% and Macro-F1 = 85.82%.
- Traditional ML: Best result was SVM with TF-IDF (Accuracy = 84%, Macro-F1 = 71%).
- LLMs: Showed mixed results. While some few-shot settings improved performance (e.g., Claude-sonnet-4.5), Arabic open-source LLMs (Fanar, ALLaM) struggled significantly in zero-shot settings.
Class-Specific Insights:
- All models performed well on Neutral and Polite classes.
- The Impolite class remained the most challenging. MARBERT excelled here (F1 = 0.81), likely due to its pretraining on social media data containing dialectal and emotionally charged language.
Error Analysis:
- Dominant Error: Neutral Bias. Models frequently misclassified both polite and impolite texts as neutral. This was most severe in LLMs (up to 48% of errors) and least in Transformers (24%).
- Specific Failure Modes:
  - Lexical Markers: Failure to detect explicit cues.
  - Ambiguity: Confusion over religious expressions (e.g., "Subhan'Allah" used sarcastically).
  - Context: Literal interpretation of culturally specific idioms or implied criticism.
  - Dialect: Poor handling of Egyptian and Maghrebi dialects due to limited pretraining coverage.

5. Key Contributions

ADAB Dataset: The first large-scale (10k samples), multi-domain, multi-dialect Arabic politeness dataset with three-way classification and 16 fine-grained linguistic features.
Comprehensive Benchmark: Establishes baseline performances for 40 models, proving that Arabic-specific fine-tuned transformers are superior to generic LLMs and traditional ML for this task.
Linguistic & Sociopragmatic Analysis: Provides a detailed error analysis highlighting the specific challenges of Arabic politeness, including the "Neutral Bias," the difficulty of detecting implicit sarcasm, and the critical need for dialectal diversity in training data.

6. Significance and Future Work

Impact: ADAB enables the development of culturally aware NLP systems for Arabic, essential for applications like customer service chatbots, social media moderation, and educational tools.
Limitations: The dataset suffers from class imbalance (dominance of neutral text) and uneven dialectal coverage (underrepresentation of Egyptian and Maghrebi varieties).
Future Directions: The authors plan to release the dataset and guidelines to the community. Future work will focus on balancing dialectal representation, incorporating conversational context (interlocutor metadata), and expanding to formal registers.

In conclusion, the paper demonstrates that while current NLP models can detect Arabic politeness, domain-specific pretraining and fine-tuning are essential to overcome the unique morphological and cultural complexities of the language, with MARBERT currently setting the state-of-the-art baseline.