💬 NLP

Curriculum Learning and Pseudo-Labeling Improve the Generalization of Multi-Label Arabic Dialect Identification Models

This paper addresses the scarcity of multi-label Arabic dialect data by constructing a new dataset via GPT-4o and dialect acceptability classifiers, then demonstrates that training a BERT-based model with curriculum learning significantly improves generalization, achieving a macro F1 of 0.69 compared to the previous state-of-the-art of 0.55.

Original authors: Ali Mekky, Mohamed El Zeftawy, Lara Hassan, Amr Keleg, Preslav Nakov

Published 2026-02-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ali Mekky, Mohamed El Zeftawy, Lara Hassan, Amr Keleg, Preslav Nakov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to recognize different regional accents in Arabic, like the difference between someone from Cairo, someone from Riyadh, and someone from Beirut.

For a long time, researchers treated this like a game of "One or None." They told the robot: "If this sentence sounds Egyptian, it's Egyptian. If it doesn't, it's NOT Egyptian." They assumed every sentence belonged to exactly one country.

But here's the problem: Language is messy. A sentence like "Make sure to count me in" might sound perfectly natural to speakers in Egypt, Jordan, Sudan, and Yemen all at the same time. It's not just Egyptian; it's a shared phrase. By forcing the robot to pick only one country, we were ignoring the reality that many sentences are multi-dialectal.

This paper is about teaching the robot to say, "Actually, this sentence works in Egypt AND Jordan AND Sudan."

Here is how they did it, broken down into simple steps:

1. The Problem: The "Wrong Negative" Trap

The researchers tried to use old data (where every sentence was labeled with just one country) to teach the robot about multiple countries. They ran into a snag.

Imagine you are teaching a student to identify "French" words. You show them a list of words that are definitely French (Positive) and a list of words that are definitely not French (Negative).

The Trap: You tell the student, "This word is not French because it's German."
The Reality: But that "German" word might actually be a loanword that French people use every day!
The Result: The student gets confused. They think, "Wait, if I see this word, I should say 'Not French,' but it sounds French to me."

The paper found that when they tried to use old data, the "Negative" examples (sentences labeled as "not this dialect") were full of sentences that actually were acceptable in that dialect. The robot was being taught to hate words it should have liked.

2. The Solution: The "Super-Teacher" and the "Panel of Judges"

Since they couldn't find enough human experts to label thousands of sentences as "Egyptian, Jordanian, AND Sudanese" (it's too expensive and slow), they built a hybrid team to create the labels automatically:

The AI Super-Teacher (GPT-4o): They asked a very advanced AI to read a sentence and guess, "Does this sound like it could be spoken in Egypt? Yes/No. Jordan? Yes/No..."
The Panel of Judges (Binary Classifiers): They also trained 18 smaller, specialized robots. Each one is an expert in only one country. One knows only Egypt, one knows only Iraq, etc.

The Magic Trick (Aggregation):
They realized each method had a weakness. The AI Super-Teacher was sometimes too generous (saying "Yes" to everything), while the Panel of Judges was sometimes too strict (saying "No" to things that were actually okay).

The Fix: They combined them. For very clear sentences (like pure Modern Standard Arabic or very strong local slang), they trusted the strict Judges. For the "gray area" sentences (the tricky ones that sound like a mix), they trusted the AI Super-Teacher.
The Result: A massive, high-quality dataset where sentences are correctly labeled with all the dialects they fit into.

3. The Training Strategy: "Curriculum Learning"

Now that they had the data, they had to teach the main robot (called LAHJATBERT). But if you throw a student into a hard exam immediately, they fail. You have to start easy.

They used a strategy called Curriculum Learning, which is like a video game with levels:

Level 1 (Easy): They showed the robot sentences that were clearly one dialect or very standard Arabic.
Level 2 (Medium): They introduced sentences that were a mix of two dialects.
Level 3 (Hard): Finally, they showed the robot the most confusing sentences that could fit into five or six different dialects.

They did this in two ways:

By Complexity: Starting with simple sentences and moving to complex ones.
By "Dialectness": Starting with sentences that sound very "standard" and slowly moving to sentences that sound very "local."

4. The Result: A Smarter Robot

The robot trained with this new method (LAHJATBERT) became much better at its job.

Before: The best robot got about 55% accuracy on the "multi-dialect" test.
After: Their new robot got 69% accuracy.

The Big Picture

Think of this like upgrading a map.

Old Map: Showed cities as isolated islands. If you were in a border town, the map didn't know which country you were in.
New Map: Shows the overlapping zones. It understands that a person in a border town might speak both languages, and that's okay.

By admitting that language overlaps and by using a smart mix of AI tools and a "step-by-step" training plan, the researchers built a system that understands the beautiful, messy reality of how people actually speak Arabic.

1. Problem Statement

Arabic Dialect Identification (ADI) has traditionally been framed as a single-label classification task, where a sentence is assigned to exactly one country-level dialect. However, linguistic reality in the Arab world is complex: a single utterance can often be acceptable and natural to speakers of multiple dialects simultaneously (dialectal overlap).

Recent benchmarks, such as NADI2024, have shifted toward Multi-Label Arabic Dialect Identification (MLADI), allowing sentences to be tagged with multiple acceptable dialects. A critical bottleneck exists: while test and development sets in NADI2024 have multi-label annotations, the training data remains single-labeled (based on user geo-location).

The Core Challenge:
Repurposing single-label datasets for multi-label training creates a "weakly-supervised" problem. The primary difficulty lies in negative sample selection. In a standard binary classifier for a specific dialect (e.g., Egyptian), samples from other countries are treated as "negative" (unacceptable). However, many of these "negative" samples are actually acceptable in the target dialect due to overlap. Treating them as negatives introduces systematic noise and hinders model generalization.

2. Methodology

The authors propose a three-stage framework to address the data mismatch and improve model performance:

A. Analysis of Training Dynamics

The authors first analyzed the behavior of binary dialect acceptability classifiers trained on single-label NADI data. They utilized training dynamics metrics (Confidence, Variability, and Correctness) to identify samples that were likely mislabeled.

Finding: A significant portion of samples labeled as "negative" (from other countries) exhibited low correctness scores during training, indicating the model struggled to classify them as negative because they were linguistically acceptable in the target dialect.
Validation: Manual annotation by native speakers confirmed that many "low-correctness negative" samples were indeed acceptable in the target dialect.

B. Construction of a Pseudo-Labelled Multi-Label Dataset

To overcome the lack of large-scale multi-label training data, the authors constructed a new dataset by aggregating pseudo-labels from two heterogeneous sources:

Binary Dialect Acceptability Classifiers: 18 independent classifiers (one per country) trained on balanced NADI 2023 data. To ensure precision, negative samples were strictly filtered (e.g., only using geolocated samples from non-neighboring regions with high dialectness scores).
GPT-4o Annotations: The LLM was prompted to independently assess the acceptability of each sentence across all 18 dialects.

Aggregation Strategy (Hybrid Approach):
The authors analyzed performance across the Arabic Level of Dialectness (ALDi) spectrum:

Low ALDi (MSA) & High ALDi (Strong Dialect): Binary classifiers showed higher precision.
Intermediate ALDi (Ambiguous/Overlap): GPT-4o showed higher recall and F1.
Final Dataset: They aggregated predictions by using Binary Classifier labels for extreme ALDi values ( $<0.11$ or $>0.77$ ) and GPT-4o labels for the intermediate range ( $0.11 \leq ALDi \leq 0.77$ ).

C. Curriculum Learning (CL) Strategies

To train the LAHJATBERT model (a fine-tuned MARBERT architecture), the authors introduced two curriculum learning strategies to handle the dataset's inherent bias toward low-cardinality labels (sentences with few dialects):

Cardinality-Based CL: Samples are ordered by the number of active labels (cardinality). The model starts with easy, low-cardinality examples and progressively encounters higher-cardinality (more ambiguous) examples.
ALDi-Based CL: Samples are ordered by their ALDi score. The model starts with clear MSA or strong dialect examples and moves toward intermediate, ambiguous examples.

Mechanism: In each stage, the model is trained on the current "hard" bucket plus an equal number of samples from all previously introduced buckets to maintain exposure to simpler cases.

3. Key Contributions

Diagnostic Analysis: Provided an in-depth analysis demonstrating that the failure of single-label data in MLADI tasks stems primarily from the misclassification of "negative" samples that are actually acceptable in multiple dialects.
Pseudo-Labeling Framework: Developed a novel method to construct a large-scale multi-label training dataset by combining the precision of binary classifiers with the recall of GPT-4o, guided by ALDi scores.
LAHJATBERT & Curriculum Learning: Introduced a family of BERT-based multi-label models trained with curriculum learning strategies. This approach effectively mitigates the low-cardinality bias and improves generalization to ambiguous dialectal overlaps.

4. Results

The models were evaluated on the NADI2024 development set and the MLADI test set (1,000 sentences, 11 dialects).

Performance on Test Set:
- The best variant (LAHJATBERT + ALDi-based Curriculum) achieved a Macro-F1 of 0.6904.
- This significantly outperformed the previous state-of-the-art NADI2024 baseline (Macro-F1 ~0.47–0.55) and other systems like Aya-32B and Elyadata.
Impact of Curriculum Learning:
- Curriculum learning consistently improved Recall (from ~69% to ~76%) at the cost of a slight drop in Precision.
- The ALDi-based curriculum was most effective for overall F1, while Cardinality-based CL yielded the highest Recall.
Comparison: The proposed system surpassed larger multilingual models (e.g., Aya-32B) and specialized Arabic models, proving the efficacy of the data construction and training strategy over simply scaling model size.

5. Significance and Conclusion

This work addresses a fundamental gap in Arabic NLP: the transition from single-label to multi-label dialect identification. By demonstrating that negative sample selection is the primary bottleneck in repurposing single-label data, the authors offer a robust solution using pseudo-labeling and curriculum learning.

The study highlights that:

Data Quality > Model Size: Careful construction of supervision signals (hybrid pseudo-labels) and training schedules (curriculum) yields better results than using massive pre-trained models alone.
Handling Ambiguity: Explicitly modeling dialectal overlap is crucial for realistic Arabic NLP applications.
Generalization: The proposed methods generalize well to unseen test data, setting a new benchmark for MLADI tasks.

The authors release their code and derived labels (though not the raw tweet text due to licensing) to facilitate reproducibility and future research in multi-dialect identification.