Technical Summary: Adapting Reinforcement Learning with Chain-of-Thought Supervision for Explainable Detection of Hateful and Propagandistic Memes

1. Problem Statement

Hateful and propagandistic memes exploit the interplay between images and text to convey harmful intent that neither modality reveals in isolation. While Multimodal Large Language Models (MLLMs) have advanced vision-language understanding, their application to meme content moderation remains underexplored, particularly regarding "thinking-based" models that generate explicit intermediate reasoning steps. Existing approaches often lack systematic integration of Chain-of-Thought (CoT) supervision, fine-grained labeling, and Reinforcement Learning (RL) for this specific domain. Furthermore, standard Supervised Fine-Tuning (SFT) offers limited control over the balance between prediction correctness and the faithfulness of generated explanations. The paper addresses the gap in applying RL-based post-training to optimize both classification performance and explanation quality for hateful and propagandistic meme detection across different languages (English and Arabic) and tasks (binary hate detection and multi-class propaganda detection).

2. Methodology

The proposed approach employs a multi-stage post-training pipeline grounded in CoT supervision and Group Relative Policy Optimization (GRPO).

2.1. Data Construction and Annotation

• Datasets: The study utilizes the Hateful Memes (FHM) benchmark (English, binary) and the Arabic Propagandistic Memes (ArMeme) dataset (Arabic, four-class: propaganda, not-propaganda, not-meme, other).
• Fine-Grained Annotations: For FHM, existing public annotations for protected categories and attack types are used. For ArMeme, where no fine-grained propaganda annotations existed, the authors developed a multi-LLM annotation pipeline. This involves two independent models (GPT-4.1 and Llama-4-Scout) generating structured propaganda technique labels and rationales, which are then consolidated by a third model (Gemini-3-Pro).
• Chain-of-Thought Distillation (CoTD): A strong teacher model (GPT-4.1) generates step-by-step reasoning traces conditioned on the meme image, text, and labels. These traces are used as supervision targets during training but are not exposed during inference to prevent label leakage.
• Unlabeled Data: For self-supervised experiments, the authors constructed unlabeled pools by aggregating public meme datasets (for FHM) and crawling social media (for ArMeme), filtering for duplicates and non-memes.

2.2. Training Pipeline

The methodology follows a three-stage process:

1. SFT Warm-up: The MLLM is initialized via Supervised Fine-Tuning on gold labels, explanations, and distilled reasoning traces. Variants include training with binary labels only, adding fine-grained labels, and incorporating distilled CoT.
2. Supervised GRPO: The model undergoes RL optimization using GRPO. Unlike PPO, GRPO does not require a learned value network; instead, it samples a group of $K$ completions and uses the group mean reward as a baseline.
   • Composite Reward Function: The reward $R(y)$ combines five components:
     • $R_{fmt}$: Format compliance (ensuring the output structure includes reasoning, label, and explanation).
     • $R_{lbl}$: Label correctness.
     • $R_{exp}$: Explanation length regularization (targeting ~100 words).
     • $R_{think}$: Thinking-length regularization. This penalizes overly short CoT traces (below a threshold $L_{min}$) to prevent "reward hacking" (e.g., empty reasoning) without incentivizing verbosity.
     • $R_{met}$: Semantic similarity (METEOR score) to the gold rationale.
3. Self-Supervised GRPO: To leverage unlabeled data, the model is further trained using consensus-based pseudo-labels. Samples are selected based on model disagreement (partial agreement across multiple temperature-sampled predictions). A majority-vote mechanism assigns pseudo-labels, which are then used to compute a reward signal alongside structural rewards.

3. Key Contributions

1. Cross-Task and Cross-Lingual Generalization: The study extends GRPO-based post-training from a single English binary dataset to a complex Arabic multi-class propaganda detection task, introducing a dual-annotator pipeline for fine-grained labeling in a previously unlabeled setting.
2. Comprehensive Empirical Analysis: The paper provides extensive ablation studies analyzing the interplay between CoT supervision, fine-grained labeling, and RL optimization across unimodal, zero-shot, and fine-tuned baselines.
3. Stabilized RL Optimization: The introduction of a threshold-based thinking-length reward ($R_{think}$) effectively mitigates reward hacking (shortening reasoning traces) while maintaining balanced per-class performance.
4. Self-Supervised GRPO: The work explores the viability of consensus-based pseudo-labeling for RL training without human annotations, demonstrating its effectiveness in in-domain, multi-class settings while highlighting limitations in binary, distribution-mismatched scenarios.

4. Results

Experiments were conducted on FHM and ArMeme benchmarks using Qwen3-VL-8B-Thinking as the base model.

• Performance Gains:
  • FHM: The proposed approach achieved 82.0% accuracy and 0.80 macro-F1, improving upon the previous state-of-the-art (79.9% accuracy, 0.79 macro-F1) by +2.1% accuracy and +0.01 macro-F1.
  • ArMeme: The approach achieved 0.612 macro-F1 (with self-supervised GRPO), representing a +7.6 point improvement over prior work (0.536) and a +6.1 point improvement over the original benchmark.
• Comparison to Baselines:
  • The generative RL-based approach outperformed strong sequence-classification baselines (e.g., Gemma-3-12B-IT, Qwen3-VL-8B-Instruct) in macro-F1 on ArMeme (0.612 vs. 0.594) while providing interpretable explanations.
  • Unimodal baselines (text-only or image-only) significantly underperformed the multimodal reasoning approach, confirming the necessity of cross-modal interaction.
• Ablation Findings:
  • Fine-Grained Supervision: Adding fine-grained labels during SFT improved macro-F1 by +8.0 points on ArMeme (0.43 to 0.51).
  • CoT Distillation: Distilled CoT provided additional gains during SFT and served as a critical initialization for GRPO.
  • SFT Warm-up: Cold-start GRPO (without SFT) resulted in degenerate performance on ArMeme (0.332 macro-F1), proving SFT initialization is critical for complex tasks.
  • Self-Supervised GRPO: Improved ArMeme macro-F1 by +1.5 points (0.597 to 0.612) but slightly reduced FHM performance (-1.0 point), attributed to distribution mismatch in the FHM unlabeled pool and majority-vote bias in binary classification.

5. Significance and Claims

The paper claims to be, to the best of the authors' knowledge, the first systematic study of GRPO in multimodal reasoning under cross-lingual, fine-grained, and self-supervised settings.

• Methodological Insight: The study demonstrates that RL-based post-training with composite rewards, specifically when combined with thinking-length regularization, can jointly optimize classification accuracy and explanation quality in a way that standard SFT cannot.
• Scalability of Synthetic Supervision: The success of the multi-LLM annotation pipeline suggests that synthetic, consensus-based fine-grained labels can serve as a scalable proxy for human annotation in content moderation tasks.
• Conditions for Self-Supervision: The work highlights that self-supervised GRPO is effective when unlabeled data is in-domain and the label space is diverse enough to prevent consensus-induced bias (as seen in the four-class ArMeme task), but less effective when distribution shifts or binary majority biases dominate (as seen in FHM).
• Generative vs. Discriminative: The results suggest that generative reasoning models offer more balanced per-class performance and actionable explanations compared to traditional sequence-classification models, making them suitable for practical content moderation where interpretability is required.

The authors release all code, data extensions, and evaluation resources to facilitate further research in this domain.