💬 NLP

Yor-Sarc: A gold-standard dataset for sarcasm detection in a low-resource African language

This paper introduces Yor-Sarc, the first gold-standard dataset for sarcasm detection in the low-resource African language Yorùbá, featuring 436 culturally annotated instances with high inter-annotator agreement to advance semantic interpretation and NLP research in the region.

Original authors: Toheeb Aduramomi Jimoh, Tabea De Wille, Nikola S. Nikolov

Published 2026-02-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Toheeb Aduramomi Jimoh, Tabea De Wille, Nikola S. Nikolov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking through a bustling marketplace in West Africa. You hear someone say, "Oh, wonderful, it's raining again! Just what I needed to ruin my dry clothes."

In English, you'd instantly know they are being sarcastic. They don't actually think the rain is wonderful; they are saying the opposite of what they mean to express frustration. But imagine if you didn't speak the language, or if you were an AI trying to understand it. You might think, "Wow, they really love the rain!" and get it completely wrong.

This is the problem the paper "Yor-Sarc" tries to solve, but specifically for Yorùbá, a language spoken by over 50 million people in Nigeria and beyond.

Here is the story of how they built a "training manual" for computers to understand sarcasm in Yorùbá, explained simply.

1. The Missing Puzzle Piece

For a long time, computers have been great at understanding English sarcasm. They have millions of examples to learn from. But for African languages like Yorùbá, it's like trying to build a house without bricks. There are very few examples of "sarcastic Yorùbá" written down for computers to study.

The researchers realized that if they want computers to understand the true feelings of Yorùbá speakers (not just the literal words), they needed to create the first-ever "Gold Standard" dataset. Think of this dataset as the ultimate answer key for a test on sarcasm.

2. Gathering the "Snippets"

The team went out and collected 436 short snippets of text. Where did they get them?

The News: From BBC News Yorùbá (the formal, edited stuff).
The Chat: From social media like Instagram, X (Twitter), and Facebook (the messy, real-life stuff).
The People: They even asked regular people to write down sarcastic things they heard in real life.

They made sure every sentence was written correctly with all the special tone marks and accents that make Yorùbá unique (since tone changes meaning in this language, just like how saying "cool" with a flat voice vs. an excited voice changes the vibe).

3. The "Three Wise Judges"

This is the most important part. To make sure the "answer key" was correct, they didn't just ask one person. They hired three native speakers of Yorùbá.

Imagine a courtroom where three judges have to decide if a statement is a joke or serious.

The Rule: They had to read the sentence and decide: "Is this person being sarcastic?" (Yes or No).
The Secret Sauce: They didn't just use a dictionary. They used their cultural intuition. They knew the local jokes, the history, and the specific way Yorùbá people use humor.

4. The "Magic" Agreement

Usually, when you ask three people to judge something subjective like sarcasm, they often disagree. One might say, "That's funny!" while another says, "No, that's serious."

But here is the amazing part: These three judges agreed almost perfectly.

83% of the time, all three judges said "Yes" or "No" together.
Two of the judges agreed 94% of the time.

To put this in perspective: In the world of English sarcasm research, getting that level of agreement is like hitting a home run in baseball. The researchers achieved a "home run" in a language that had never been studied this way before. It proves that when you use native speakers who understand the culture, you can teach computers to spot sarcasm even in complex, tonal languages.

5. Handling the "Maybe" Cases

What about the 17% of the time they didn't agree?
Instead of forcing a "Yes" or "No" and throwing away the confusion, the researchers kept the disagreement. They created "Soft Labels."

Think of it like a dimmer switch on a light.

If all three say "Sarcastic," the light is 100% bright.
If two say "Sarcastic" and one says "No," the light is 66% bright.
If one says "Sarcastic" and two say "No," the light is 33% bright.

This teaches the computer: "Hey, this one is a bit tricky. Don't be 100% sure; be a little uncertain." This makes the AI smarter and more human-like.

Why Does This Matter?

Before this paper, if you built an AI to analyze Yorùbá tweets, it would miss the jokes and misunderstand the anger. It would think a sarcastic insult was a compliment.

Yor-Sarc is the foundation. It's the first step to:

Better AI: Helping computers understand the real meaning behind Yorùbá words.
Cultural Respect: Showing that African languages are complex and rich enough to have their own high-quality research tools.
Future Growth: Now that they have this "Gold Standard," other researchers can use it to build better tools for all of Africa, not just for Yorùbá.

In a nutshell: The researchers built the first-ever "Sarcasm Dictionary" for Yorùbá by getting three native experts to agree on 436 examples. They proved that with the right cultural guidance, even the trickiest human emotions can be taught to machines.

1. Problem Statement

Sarcasm detection is a critical yet challenging sub-task in Natural Language Processing (NLP) because it requires models to resolve the disparity between literal and intended meaning, relying heavily on pragmatic cues, cultural context, and literary knowledge.

The Low-Resource Gap: While significant progress has been made in high-resource languages (e.g., English), low-resource languages, particularly African languages, suffer from a severe scarcity of annotated datasets.
The Yorùbá Void: Yorùbá, a tonal Niger-Congo language spoken by over 50 million people, lacks any publicly documented gold-standard corpus for sarcasm detection. Existing resources for African languages (e.g., AfriSenti) focus on coarse-grained sentiment analysis, ignoring fine-grained figurative phenomena like sarcasm.
The Challenge: The absence of high-quality, culturally grounded data hinders the development of sarcasm-aware sentiment systems and excludes Yorùbá pragmatics from the global scientific understanding of sarcasm.

2. Methodology

The authors introduced Yor-Sarc, the first manually annotated, gold-standard sarcasm dataset for the Yorùbá language.

Data Collection

Corpus Size: 436 short text instances.
Sources: A multi-source approach was used to balance formal and informal registers:
- BBC News Yorùbá: 65.4% (285 instances) – professionally edited, context-rich.
- Social Media: 28.5% (124 instances) from Instagram, X/Twitter, Facebook, and YouTube.
- Crowdsourcing: 3.9% obtained via an ethically approved online survey to capture face-to-face conversational contexts.
Orthography: All text is in Standardised Yorùbá (SY) with proper tone and diacritic marks.

Annotation Framework

Annotators: Three native Yorùbá speakers with linguistic expertise and fluency in diverse dialects (e.g., Standard, Ifè, `Ijèbú).
Protocol:
- Annotators worked independently without consultation.
- Guidelines were developed through an iterative pilot study (20 examples) and refined to account for cultural nuances.
- Labeling: Binary classification (Sarcastic vs. Non-sarcastic) based on whether the statement conveys meaning contrary to its literal interpretation.
Gold Standard Derivation:
- Hard Labels: Derived via majority voting (2 out of 3).
- Soft Labels: Preserved for the 16.7% of cases where annotators disagreed (2-1 split), calculated as the proportion of sarcastic votes ($0.0, 0.333, 0.667, 1.0$). This allows for uncertainty-aware modeling rather than forcing artificial consensus.

Evaluation Metrics

The study employed a comprehensive Inter-Annotator Agreement (IAA) analysis:

Pairwise Agreement: Cohen's $\kappa$ (correcting for chance) for all pairs.
Multi-Rater Agreement: Fleiss' $\kappa$ for overall consistency.
Uncertainty Quantification: Binary entropy calculated for instances to identify "hard cases."
Bias Analysis: Measured systematic deviations in labeling thresholds among annotators.

3. Key Contributions

First Gold-Standard Dataset: The creation of Yor-Sarc, the first publicly available sarcasm corpus for Yorùbá, filling a critical gap in African NLP resources.
Culturally Grounded Protocol: Documentation of an annotation framework specifically tailored to Yorùbá sarcasm, incorporating dialectal diversity and cultural context, which serves as a blueprint for other low-resource African languages.
Advanced IAA Analysis: A rigorous analysis of agreement patterns, including the preservation of soft labels for ambiguous cases, providing a deeper understanding of semantic uncertainty in figurative language.
Benchmarking: Establishing a new benchmark for sarcasm detection in low-resource, tonal languages.

4. Results

The dataset achieved remarkably high levels of agreement, surpassing many existing benchmarks for high-resource languages.

Overall Agreement:
- Fleiss' $\kappa$ : 0.7660 (Substantial).
- Average Pairwise Cohen's $\kappa$ : 0.7671.
- Unanimous Consensus: 83.3% (363 out of 436 instances) received a 3-0 agreement.
Pairwise Performance:
- Best Pair (A1–A2): Achieved $\kappa = 0.8743$ (93.81% raw agreement), categorized as "Almost Perfect."
- Other Pairs: A1–A3 ( $\kappa = 0.7539$ ) and A2–A3 ( $\kappa = 0.6732$ ) both achieved "Substantial" agreement.
Comparison with Benchmarks:
- Yor-Sarc's average $\kappa$ (0.7671) exceeds all reported English sarcasm datasets (e.g., González-Ibáñez et al. 2011: 0.56–0.62; Riloff et al. 2013: 0.67).
- The best pair's performance (0.8743) exceeds even the high-confidence subset of Bamman and Smith (2015, $\kappa = 0.81$ ).
Annotator Behavior:
- Disagreements were not due to inconsistent guideline application but rather interpretive threshold differences.
- Annotator A2 was the most "liberal" (45.87% sarcastic labels), while A3 was the most "conservative" (30.96%).
- Confusion matrices revealed that disagreements were often symmetric (random) between similar-threshold annotators but asymmetric between those with divergent thresholds.

5. Significance

Validation of Native Expertise: The study demonstrates that native speakers with shared cultural backgrounds can achieve "almost perfect" agreement on subjective semantic tasks in low-resource, tonal languages, provided the annotation protocol is culturally informed.
Handling Ambiguity: By preserving soft labels for the 16.7% of ambiguous cases, the dataset supports uncertainty-aware modeling. This aligns with modern NLP trends that treat disagreement as a signal of inherent semantic ambiguity rather than noise.
Resource Expansion: Yor-Sarc enables the development of sarcasm-aware sentiment analysis tools for over 50 million Yorùbá speakers, preventing the misinterpretation of satirical content as genuine sentiment.
Future Directions: The dataset lays the groundwork for expanding sarcasm detection to other African languages and advancing research into the scalar nature of figurative language interpretation.

In conclusion, Yor-Sarc represents a significant leap forward in African NLP, proving that high-quality, culturally nuanced datasets can be constructed even for low-resource languages, thereby enabling more robust and reliable AI systems for diverse global populations.