Technical Summary: LLM-Driven Data Augmentation and Fine-Tuned Transformers vs. Few-Shot LLMs for Arabic Propaganda Technique Detection

Problem Statement

The detection of propaganda techniques in Arabic social media is a critical task for media literacy and platform governance, yet it remains significantly under-explored compared to English. The field faces three primary challenges:

1. Data Scarcity and Imbalance: The only available fine-grained benchmark, the WANLP 2022 corpus, contains only 930 annotated tweets across 17 propaganda technique labels. The training split (607 entries) suffers from severe class imbalance, where the most frequent label ("Loaded Language") appears 446 times while the rarest ("Presenting Irrelevant Data") appears only once.
2. Morphological Complexity: Arabic is a morphologically rich language with complex root-pattern inflections. Standard data augmentation techniques designed for English (e.g., synonym replacement, random token insertion/deletion) often fail in Arabic, producing ungrammatical outputs that degrade model performance.
3. Model Paradigm Uncertainty: It is unclear whether fine-tuned pre-trained transformers or few-shot Large Language Model (LLM) prompting is more effective for this specific multi-label, low-resource task. Furthermore, the utility of adding explicit linguistic features (POS, NER) to transformer models in this context remains uninvestigated.

Methodology

The authors propose a complete pipeline addressing data scarcity and model selection, structured into two main phases:

1. Data Augmentation Strategies

To expand the training set from 607 to 4,191 unique entries, three distinct augmentation strategies were systematically compared and manually filtered:

• LLM Synonym Replacement: Using the Cohere command-r model to generate grammatically correct Arabic synonyms for selected tokens. This yielded a 68.4% acceptance rate.
• Pre-trained Model Augmentation: A combination of back-translation (Arabic→English→Arabic via Helsinki-NLP models) and text generation using araGPT2. These achieved acceptance rates of 89.6% and 84.1%, respectively.
• Few-Shot Dataset Generation: Using GPT-4.1 to synthesize entirely new propaganda examples based on style guides (seed examples) for specific techniques. This strategy achieved the highest quality with a 98.5% acceptance rate and successfully enriched multi-label diversity.

Note: Traditional methods like nlpaug synonym replacement and random token operations were tested but rejected due to error rates exceeding 20%, primarily caused by morphological agreement failures.

2. Model Benchmarking

The augmented dataset was evaluated across two paradigms:

• Fine-Tuned Deep Learning: 12 experiments were conducted using three architectures: BERT (CAMeLBERT), CNN+LSTM+Attention, and RNN+LSTM+Attention. These were tested on both the original and augmented datasets, with and without explicit Part-of-Speech (POS) and Named Entity Recognition (NER) features.
• Few-Shot LLM Prompting: Four LLMs (GPT-5, DeepSeek-V3.2, LLaMA-3.3-70b, and Qwen3-32b) were evaluated using a zero-shot/few-shot prompting approach with 15 manually selected examples covering all 17 technique labels.

Training Configuration: Models utilized Adaptive Focal Loss (for CNN/RNN) or Hybrid Focal/Dice Loss (for BERT) to address class imbalance. Post-processing included threshold optimization and hybrid rule-based boosting for rare classes.

Key Contributions

1. Systematic Augmentation Comparison: The paper provides the first systematic comparison of LLM-based generation versus traditional augmentation for Arabic propaganda detection. It establishes that GPT-4.1 few-shot generation yields the highest quality data (98.5% acceptance), while traditional methods often fail due to Arabic morphology.
2. Comprehensive Benchmarking: It presents the first direct comparison between fine-tuned deep learning models and few-shot LLMs on the WANLP 2022 benchmark, covering 16 distinct experimental configurations.
3. New State-of-the-Art: The authors establish a new state-of-the-art system using BERT fine-tuned on the augmented dataset, surpassing the WANLP 2022 shared task winner.
4. Counter-Intuitive Findings on Features: The study reveals that adding POS and NER features to pre-trained transformers unexpectedly degrades performance, challenging the assumption that explicit linguistic features always aid Arabic NLP tasks.

Results

• Best Overall System: BERT fine-tuned on the augmented dataset (without linguistic features) achieved F1-Micro = 0.660 and F1-Macro = 0.470.
  • This surpasses the WANLP 2022 winner (Hussein et al., 2022) which scored F1-Micro = 0.649 and F1-Macro = 0.185.
  • The improvement in F1-Macro (+28.5 points) indicates significantly better detection of rare propaganda techniques.
• Impact of Augmentation: Data augmentation was the single most significant factor. For BERT, augmentation improved F1-Micro from 0.57 to 0.66 and F1-Macro from 0.20 to 0.47.
• Impact of Linguistic Features: Adding POS/NER features consistently reduced performance. For BERT on the original dataset, F1-Micro dropped from 0.57 to 0.23 when features were added.
• LLM Performance: Among few-shot LLMs, DeepSeek-V3.2 performed best with F1-Micro = 0.54 and F1-Macro = 0.32. While competitive for frequent techniques, LLMs generally underperformed compared to fine-tuned BERT, particularly on rare classes.
• Architecture Comparison: While BERT generally outperformed CNN and RNN hybrids, the RNN+LSTM+Attention model with features achieved the highest F1-Macro (0.49) in one specific configuration, though at the cost of a lower F1-Micro (0.48).

Significance and Claims

The paper claims that data augmentation is the single most important factor in building effective Arabic propaganda detection systems in low-resource settings. The study demonstrates that:

1. Fine-tuning beats few-shot prompting: For fine-grained, multi-label tasks with severe class imbalance, fine-tuning on augmented data is superior to few-shot LLM prompting.
2. Morphological sensitivity is critical: Standard NLP augmentation strategies are insufficient for Arabic; LLM-driven generation or careful back-translation is required to maintain grammatical integrity.
3. Feature engineering caution: Practitioners should not assume that combining syntactic features (POS/NER) with pre-trained transformers yields gains for Arabic social media text; in this context, the noise introduced by imperfect taggers and redundancy with transformer embeddings appears detrimental.

The authors conclude that their pipeline provides practical guidance for researchers building Arabic NLP resources, emphasizing that high-quality data augmentation can close the performance gap between deep learning and LLMs, even in highly imbalanced scenarios. Future work is suggested to investigate direct fine-tuning of LLMs on the augmented dataset and cross-lingual transfer.