💬 NLP

DariMis: Harm-Aware Modeling for Dari Misinformation Detection on YouTube

This paper introduces DariMis, the first manually annotated dataset of 9,224 Dari-language YouTube videos, and demonstrates that a specialized ParsBERT model using a pair-input encoding strategy effectively detects misinformation by leveraging the structural coupling between information type and harm levels.

Original authors: Jawid Ahmad Baktash, Mosa Ebrahemi, Mohammad Zarif Joya, Mursal Dawodi

Published 2026-03-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jawid Ahmad Baktash, Mosa Ebrahemi, Mohammad Zarif Joya, Mursal Dawodi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a massive, bustling global marketplace. For decades, the "security guards" (AI tools) who check for fake news have been very good at their jobs in the English-speaking sections of the market. But in the Dari-speaking section of Afghanistan and Central Asia—a place where millions of people rely on YouTube for their news and health advice—there were no guards at all.

This paper introduces DariMis, a project that finally puts a security guard on duty in that Dari-speaking section. Here is the story of how they did it, explained simply.

1. The Problem: A Blind Spot in the Market

For years, researchers built tools to spot lies, but they mostly spoke English or other "rich" languages. Dari, spoken by 20–40 million people, was ignored.

The Gap: If you speak Dari, you were left vulnerable to fake news, conspiracy theories, and dangerous medical advice on YouTube with no automated help.
The Danger: It's not just about "True" vs. "False." Some lies are harmless (like a fake movie plot), but others are deadly (like fake cures for diseases). The old tools treated all lies the same, which is like treating a paper cut the same way as a broken leg.

2. The Solution: Building the "DariMis" Database

The team created the first-ever manual database of 9,224 Dari YouTube videos. Think of this as training a new security guard by showing them thousands of examples.

They didn't just label videos as "Lie" or "Truth." They used a two-dimensional scorecard:

Truth Level: Is it a Lie, a Half-Truth, or the Real Deal?
Harm Level: Is this dangerous? (Low, Medium, or High).

The Big Discovery: They found a secret link between these two scores.

The Analogy: Imagine a fruit stand.
- Real Fruit (True): Almost never poisonous.
- Fake Fruit (Misinformation): 56% of the time, it's actually poisonous (High/Medium harm).
- Half-Rotten Fruit (Partly True): This is the most common type (60% of the data). It looks real but has a bad core. It's tricky because it's mostly true, which makes it very persuasive.

Why this matters: If you build a system that catches "Lies," you automatically catch most of the "Poisonous" content. You don't need a separate test for danger; the lie detector acts as a safety filter.

3. The Secret Weapon: The "Headline vs. Body" Trick

Most AI systems read a video's title and description as one long, messy paragraph. The researchers realized this was like reading a book cover and the first page mixed together.

The Old Way: Mixing the title and description together.
The New Way (Pair-Input): They taught the AI to read the Title (the headline) and the Description (the body) as two separate but connected characters in a play.

The Analogy: Imagine a salesman shouting, "BREAKING NEWS: Miracle Cure Found!" (Title) but then whispering in the description, "Just eat some carrots." (Description).

If you read them as one block, you might miss the contradiction.
If you compare them side-by-side, the AI sees the inconsistency. The title is screaming, but the body is calm. That mismatch is the biggest clue that something is wrong.

The Result: This simple change made the AI 7% better at catching the dangerous "Lies," even though the overall score didn't change much. In safety terms, catching 7% more dangerous lies is a huge win.

4. The Language Expert vs. The Generalist

They tested two types of AI brains:

XLM-RoBERTa: A "Generalist" who speaks 100 languages but knows none of them deeply.
ParsBERT: A "Specialist" who grew up speaking Dari and Farsi.

The Result: The Specialist (ParsBERT) won. It understood the subtle dialects, the specific cultural phrases, and the nuances of Dari much better than the Generalist. It was like hiring a local detective who knows the neighborhood versus a tourist with a map.

5. The Honest Truth: It's Not Perfect

The authors are very honest about the limitations.

The "Gray Area" Problem: Sometimes, even human experts argue about whether a video is a "Lie" or a "Half-Truth." If humans can't agree 100% of the time, the AI can't either.
Missing Info: About 30% of the videos only had a title and no description. Without the "body" text to compare against the "headline," the AI had to guess, which made it less accurate.
The "Partly True" Trap: The hardest videos to catch are the ones that are mostly true but framed in a misleading way. This is the "gray area" where most of the errors happened.

Summary: Why Should You Care?

This paper is a blueprint for protecting vulnerable communities online.

It fills a gap: It gives Dari speakers their first line of defense against YouTube misinformation.
It prioritizes safety: By focusing on the "Headline vs. Body" mismatch, it catches the most dangerous lies first.
It teaches us how to build better tools: It shows that for low-resource languages, you need a language specialist, not just a general translator, and that understanding the structure of a lie (title vs. description) is more important than just reading the words.

In short, they built a smarter, more culturally aware security guard for a community that was previously left unguarded, proving that sometimes, the best way to spot a lie is to check if the headline matches the story.

1. Problem Statement

The paper addresses a critical gap in Natural Language Processing (NLP) and content moderation: the lack of automated misinformation detection tools for Dari, the primary language of Afghanistan (spoken by 20–40 million people).

The Gap: Existing research focuses heavily on high-resource languages and binary classification (True vs. False). Low-resource languages like Dari are underrepresented, leaving millions of users vulnerable to unmoderated harmful content on platforms like YouTube.
The Complexity: Real-world misinformation is rarely binary. It often involves "Partly True" content (accurate facts in misleading contexts) and varies significantly in harm potential (from trivial errors to incitement of violence). Current binary systems fail to capture these nuances or the structural link between accuracy and harm.

2. Dataset: DariMis

The authors introduce DariMis, the first manually annotated dataset for Dari-language YouTube misinformation.

Scale: 9,224 unique videos collected via the YouTube Data API (2007–2026).
Dual-Axis Annotation: Each video is labeled on two orthogonal dimensions:
1. Information Type: Misinformation, Partly True, True.
2. Harm Level: Low, Medium, High.
Distribution: The dataset is imbalanced, with "Partly True" being the majority class (60.0%), followed by Misinformation (22.6%) and True (17.4%).
Key Finding (Structural Coupling): The authors discovered a strong statistical coupling between accuracy and harm. 55.9% of Misinformation carries at least "Medium" or "High" harm, whereas only 1.0% of "True" content does. This implies that an accurate Information Type classifier can implicitly function as a harm-triage filter.

3. Methodology

The study proposes a novel modeling approach tailored to the structural nature of YouTube metadata (Title vs. Description).

Pair-Input Encoding Strategy:
- Problem with Standard Approaches: Most models concatenate the title and description into a single flat sequence, losing the structural relationship between the headline and the body.
- Proposed Solution: The authors utilize BERT's native segment-pair mechanism. The Title is fed as Segment A and the Description as Segment B. This allows the model's self-attention mechanism to explicitly learn cross-segment semantic dependencies (e.g., detecting when a sensational title contradicts a factual description).
- Handling Missing Data: For videos without descriptions (31.2% of the dataset), the model processes the title alone.
Models Evaluated:
- ParsBERT: A BERT-base model pre-trained specifically on large Farsi/Dari corpora.
- XLM-RoBERTa-base: A multilingual model pre-trained on 2.5TB of CommonCrawl text across 100 languages.
- Training: Both models were fine-tuned using a linear classification head on the [CLS] token with AdamW optimization.

4. Key Results

The experiments were conducted on a stratified test set (1,393 samples) with rigorous statistical evaluation using Bootstrap 95% Confidence Intervals (CI).

Model Performance:
- ParsBERT outperformed XLM-RoBERTa across all metrics.
- XLM-RoBERTa suffered from "majority-class collapse," achieving extremely low F1 scores for minority classes (Misinformation F1 = 0.26; True F1 = 0.03).
- ParsBERT maintained consistent performance across all three classes, achieving a Macro F1 of 72.77% and Accuracy of 76.60%.
Impact of Pair-Input Encoding (Ablation Study):
- While the overall Macro F1 difference between single-field concatenation and pair-input encoding was marginal (+0.09 pp), the Misinformation Recall saw a significant jump: +7.0 percentage points (from 60.1% to 67.1%).
- Significance: In content moderation, missing a high-harm video (False Negative) is more costly than flagging a borderline case. The pair-input approach significantly improved the detection of this safety-critical class.
Statistical Rigor:
- The authors noted that the overall performance differences between models were not statistically significant at the 95% level due to the test set size and the inherent difficulty of the task (human inter-annotator agreement $\kappa \approx 0.71$ ).
- However, they argue that the qualitative patterns (ParsBERT's stability vs. XLM-RoBERTa's collapse) and the class-specific gains in recall are practically significant for deployment.

5. Error Analysis

Semantic Attractor: The "Partly True" class acts as a semantic attractor, accounting for 56.8% of all errors. This reflects the difficulty in distinguishing between misleading framing and incomplete context.
Conservative Behavior: The model rarely predicts "True" as "Misinformation" (1.8% of errors), which is desirable for avoiding false accusations, though it slightly underestimates the severity of fringe Misinformation.
Framing Effects: The primary failure mode involves "framing effects" where factual claims are embedded in misleading narratives—a task requiring world knowledge beyond current text-only models.
Missing Descriptions: The pair-input advantage is lost for the 31.2% of videos with no description, leading to higher error rates for these samples.

6. Contributions and Significance

DariMis Dataset: A foundational resource for low-resource NLP, providing the first dual-axis (Type + Harm) annotated corpus for Dari.
Harm-Aware Modeling: Demonstrates that Information Type and Harm Level are structurally coupled, enabling classifiers to serve as implicit harm-triage filters in moderation pipelines.
Pair-Input Encoding: Proves that explicitly modeling the Title-Description relationship via BERT segments significantly improves the recall of the most dangerous class (Misinformation), even if overall accuracy gains are modest.
Language Specialization: Validates that language-specialized pre-training (ParsBERT) is superior to general multilingual models for low-resource, morphologically complex languages like Dari.
Transparent Evaluation: The paper sets a high standard for statistical reporting in NLP by using bootstrap confidence intervals to honestly report the limitations of statistical significance in low-resource settings.

Conclusion

The paper concludes that while automated detection of Dari misinformation remains challenging due to linguistic ambiguity and data scarcity, the combination of specialized pre-training and structural pair-input modeling offers a viable path forward. Future work should focus on multimodal integration (audio/visual), joint prediction of accuracy and harm, and expanding the dataset to improve statistical power. The DariMis dataset is released to the community to facilitate further research.