From FusHa to Folk: Exploring Cross-Lingual Transfer in Arabic Language Models

1. Problem Statement

Arabic presents a unique challenge for Natural Language Processing (NLP) due to its pervasive diglossia: a standardized written variety, Modern Standard Arabic (MSA/FusHa), coexists with a rich spectrum of non-standardized Dialectal Arabic (DA) varieties used in daily communication and online interaction.

• The Core Issue: Most existing Arabic Large Language Models (LLMs) are pretrained predominantly on MSA. While these models are expected to generalize to dialects, the linguistic distance between MSA and various dialects varies significantly.
• The Gap: It remains unclear whether MSA-centric models can effectively transfer knowledge to diverse dialects, or if "multi-dialect" models (trained on a mix of MSA and various dialects) suffer from negative interference, where training on multiple varieties degrades performance on specific high-resource dialects compared to specialized models.
• Research Questions:
  1. To what extent do MSA and multi-dialect models transfer to specific dialects across core NLP tasks?
  2. Does geographic proximity between a dialect and an MSA "anchor" predict transferability?
  3. Do multi-dialect models suffer from negative interference compared to specialized dialect models?

2. Methodology

The authors employ a dual-methodology framework combining functional probing and representational similarity analysis to evaluate cross-lingual transfer.

A. Experimental Setup

• Models: The study evaluates a diverse set of BERT-style encoders, including:
  • MSA-centric: CAMeLBERT-MSA.
  • Multi-dialect/Mixed: CAMeLBERT-MIX (MSA + DA + Classical Arabic) and CAMeLBERT-Multi-DA.
  • Dialect-Specific: Models for Saudi, Egyptian, Moroccan, Algerian, Omani, Lebanese, and Jordanian dialects (e.g., SaudiBERT, EgyBERT, DarijaBERT).
• Tasks: Three core NLP tasks were selected to probe different linguistic levels:
  1. Part-of-Speech (POS) Tagging: Syntactic structure.
  2. Named Entity Recognition (NER): Semantic entity extraction.
  3. Sentiment Analysis (SA): Pragmatic and semantic understanding.
• Datasets:
  • Probing: Balanced datasets for each dialect (derived from MADAR and other sources), with strict dialect identification filtering to remove noise.
  • Similarity: The MADAR corpus, providing parallel sentences in MSA and 25 city-level dialects.

B. Evaluation Techniques

1. Linear Probing:
   • Lightweight linear classifiers (logistic regression) are trained on frozen layer-wise embeddings from the pre-trained models.
   • Goal: To determine if linguistic features (POS, NER, Sentiment) are encoded in the model's representations and how easily they can be extracted.
   • Metric: Macro-F1 score on the best-performing layer.
2. Representational Similarity Analysis (RSA):
   • Uses Centered Kernel Alignment (CKA) to measure the similarity between the hidden representations of MSA models and Dialect models.
   • Scenarios:
     1. MSA model vs. Dialect model on Dialect sentences.
     2. MSA model vs. Dialect model on MSA sentences.
     3. MSA model on MSA sentences vs. Dialect model on Dialect sentences.
   • Goal: To provide an intrinsic, task-agnostic measure of how similarly the models encode Arabic input.
3. Geographic Proximity Analysis:
   • The authors treat Yemen as a geographic proxy for the MSA anchor (based on literature linking Yemeni Arabic to Classical Arabic and MSA).
   • They correlate model performance/similarity with the geographic distance of the dialect's country from Yemen.

3. Key Contributions

• Holistic Evaluation Framework: The paper integrates functional probing (task performance) with structural analysis (CKA) to provide a comprehensive view of cross-lingual transfer in Arabic.
• Quantification of Negative Interference: It provides empirical evidence that training models on multiple dialects simultaneously can lead to negative interference, particularly for high-resource dialects, challenging the assumption that "more data" always equals "better transfer."
• Geographic Continuum Validation: The study validates the dialect continuum hypothesis in the context of deep learning, demonstrating that geographic proximity is a strong predictor of transferability, but only when sufficient pretraining data is available.
• Data Volume vs. Specialization: It establishes that specialized dialect models only outperform general models when supported by substantial pretraining data; otherwise, general models often perform better due to the "curse of multilinguality" affecting low-resource dialects less severely.

4. Key Results & Findings

A. Probing Performance (Functional Transfer)

• MSA Dominance on MSA: MSA-pretrained models consistently outperform MIX and Multi-DA models on MSA tasks, suggesting that mixing dialects dilutes MSA representation quality.
• Task-Specific Transfer:
  • POS & NER: MSA models remain competitive on these structural tasks for many dialects, indicating that syntactic and entity-structure regularities are largely shared.
  • Sentiment Analysis (SA): Dialect-specific models significantly outperform MSA models on SA. This suggests that sentiment relies heavily on dialect-specific idioms, non-standard orthography, and pragmatic nuances that MSA models fail to capture.
• The "Curse of Multilinguality":
  • High-Resource Dialects (e.g., Egyptian, Saudi): Specialized models (EgyBERT, SaudiBERT) often outperform general models because they have sufficient data to learn specific nuances without interference.
  • Low-Resource Dialects (e.g., Lebanese, Jordanian): General/Multi-dialect models often outperform specialized models, as the specialized models suffer from data scarcity.

B. Representational Similarity (CKA)

• Low Absolute Similarity: Even for the most similar dialects, CKA scores between MSA and DA models rarely exceed 0.8, indicating that MSA models fail to fully capture dialect-specific nuances.
• Scenario Differences: Similarity is highest when both models process the same input type (e.g., MSA model on MSA text vs. DA model on MSA text). When inputs differ (Scenario 3), similarity drops, proving representations are not input-variant agnostic.
• Data Dependency: The correlation between CKA similarity and pretraining data volume is strong. Models with larger corpora (e.g., BERT-based) show clearer dialect continuum patterns than smaller models (e.g., AraRoBERTa variants).

C. Geographic Proximity

• Dialect Continuum: There is a clear correlation between geographic distance from the Yemen (MSA anchor) and model performance/similarity.
  • Close Proximity (e.g., Saudi, Levantine): Higher transferability and higher CKA similarity.
  • Far Proximity (e.g., Maghrebi/North African): Lower transferability and lower similarity.
• Nuance: This geographic effect is only observable in models trained on large-scale corpora. Models with limited data do not internalize the dialect continuum effectively.

5. Significance and Implications

• Challenging Assumptions: The paper refutes the assumption that MSA models can seamlessly transfer to all Arabic dialects. Transfer is disproportionate; it works well for structurally similar, geographically close dialects but fails for distant ones, especially in semantic tasks like sentiment analysis.
• Model Architecture Strategy:
  • For High-Resource Dialects: Specialized, dialect-specific models are superior, provided they have sufficient training data.
  • For Low-Resource Dialects: General multi-dialect models are preferable to avoid data scarcity issues.
  • Negative Interference: Training on all dialects simultaneously can harm performance on high-resource dialects. Future models may need dialect-specific parameters or adapters to mitigate this.
• Future Directions: The study highlights the need for better dialect-specific datasets (especially for Gulf NER) and suggests that geographic proximity can serve as a heuristic for selecting transfer sources in low-resource NLP settings.

In conclusion, the paper demonstrates that while cross-lingual transfer in Arabic is possible, it is heavily constrained by geographic distance, pretraining data volume, and task type. A "one-size-fits-all" MSA model is insufficient for the diverse Arabic linguistic landscape.