💬 NLP

Fine-tuning with Hierarchical Prompting for Robust Propaganda Classification Across Annotation Schemas

This paper introduces a new intent-focused propaganda taxonomy and the Hierarchical Prompting (HiPP) method, demonstrating that fine-tuning language models with this approach significantly improves robustness and performance across varying annotation schemas, particularly for ambiguous, low-agreement datasets.

Original authors: Lukas Stähelin, Veronika Solopova, Max Upravitelev, David Kaplan, Ariana Sahitaj, Premtim Sahitaj, Charlott Jakob, Sebastian Möller, Vera Schmitt

Published 2026-05-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lukas Stähelin, Veronika Solopova, Max Upravitelev, David Kaplan, Ariana Sahitaj, Premtim Sahitaj, Charlott Jakob, Sebastian Möller, Vera Schmitt

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a group of robots to spot propaganda in social media posts. The problem is that propaganda is tricky: it's short, messy, and often depends on why someone wrote it, not just what they wrote.

This paper is like a report card on how well different robot teachers (AI models) learned this task, and it tests three different ways of teaching them.

The Two "Textbooks" (Annotation Schemas)

First, the researchers created two different "textbooks" or rulebooks for the robots to learn from.

The "Technique" Textbook (Sahitaj et al.): This one is like a checklist of specific tricks. It says, "If you see a scary word, that's 'Fear.' If you see a name-calling, that's 'Ad Hominem'." It's very clear-cut, and the human teachers (annotators) agreed on these rules most of the time. It's easy to grade, but it might miss the bigger picture.
The "Intent" Textbook (The New One): This is the researchers' new invention. Instead of just listing tricks, they asked, "What is the goal of this post?" Is the goal to "Shift the Blame"? Is it to "Confuse the Audience"? Is it to "Justify Aggression"?
- The Catch: This is much harder. It's like asking a student to guess the teacher's motive rather than just spotting a grammar mistake. The human teachers disagreed a lot more on these labels because the motives are fuzzy and subjective.

The Three Teaching Methods

The researchers tested four different robot brains (AI models) using three different teaching strategies:

The "Zero-Shot" Method: You just hand the robot the textbook and say, "Go figure it out." The robots were terrible at this. They guessed randomly, like a student who didn't study.
The "Fine-Tuning" Method: This is the big breakthrough. The researchers showed the robots thousands of examples and let them practice until they got it right.
- The Result: This was the magic key. It turned weak, confused robots into sharp, competitive ones. The paper claims that without this practice session, the robots simply couldn't learn the task well.
The "Hierarchical Prompting" (HiPP) Method: This is a clever trick. Instead of asking the robot, "What is the main goal?" immediately, they asked it two questions in a row:
- Step 1: "What specific trick is being used here?"
- Step 2: "Okay, based on that trick, what is the main goal?"
- The Analogy: Imagine trying to identify a bird. Instead of guessing the species immediately, you first ask, "Does it have a long beak?" and then, "Based on the long beak, is it a heron?"
- The Finding: This two-step method worked best after the robots had practiced (fine-tuned), especially on the difficult "Intent" textbook where the rules were messy. It helped the robots stay organized when the answers were confusing.

The Results: Who Won?

The Best Robots: The Qwen family of robots (specifically Qwen3-14B) performed the best overall. They were the "honor students."
The Runner-Up: The Phi-4 robot was also very strong, beating the GPT-4.1-nano robot consistently.
The Lesson on Difficulty: The robots scored higher on the easy "Technique" textbook than on the hard "Intent" textbook. This proves that the new "Intent" rules are indeed more challenging and realistic, even if they are harder to learn.

The "Aha!" Moment (Error Analysis)

The researchers looked at where the robots made mistakes.

Before Practice: The robots were "keyword hunters." If they saw words like "War," "Russia," or "Nazi," they immediately screamed "PROPAGANDA!" even if the post was just a news report. They were too sensitive.
After Practice: The robots became much smarter. They stopped flagging every news report about war. They only flagged the posts that were actually trying to manipulate people.
The Confusion: When the robots did get confused, they didn't pick random answers. They confused similar goals (like mixing up "Shifting Blame" with "Distorting Reality"). This is actually good news; it means they are thinking logically, just getting the fine details wrong, much like a human would.

The Bottom Line

The paper concludes that to build a robot that can really understand propaganda, you can't just give it a rulebook and hope for the best. You have to train it (fine-tune) on real examples. Once trained, using a step-by-step thinking process (HiPP) helps it handle the messy, real-world situations where the "why" behind a message is hard to pin down.

They also released their new "Intent" textbook and the trained robots for others to use, hoping it will help researchers understand not just that propaganda is happening, but what the propagandists are trying to achieve.

Technical Summary: Fine-tuning with Hierarchical Prompting for Robust Propaganda Classification Across Annotation Schemas

Problem Statement
Propaganda detection in social media faces significant challenges due to the noisy, short nature of texts, the prevalence of informal language, and low inter-annotator agreement (IAA). Existing taxonomies are often technique-centric (focusing on rhetorical devices like "loaded language") and struggle to capture the strategic communicative intents behind propaganda. Furthermore, there is a trade-off in schema design: realistic, complex schemas capture real-world diversity but reduce annotation reliability, while simplified schemas boost agreement at the cost of ecological validity. Additionally, the interaction between annotation schema design, supervision regimes (zero-shot vs. fine-tuned), and prompting strategies remains under-explored in the context of Large Language Models (LLMs).

Methodology
The authors conducted a controlled experiment evaluating propaganda classification across three interacting dimensions: annotation schema, supervision regime, and prompting strategy.

Datasets and Schemas:
- Data Source: The study utilized the HQP benchmark dataset (30,000 English tweets from the start of the Russia-Ukraine war).
- Schema A (New Intent-Based): The authors introduced a new taxonomy grouping 17 fine-grained techniques into five high-level categories based on communicative intent (e.g., Shift Blame, Justify Aggression, Manufacture Consent). This schema was developed via an iterative process involving expert annotation, LLM-assisted harmonization, and group consensus. It yielded moderate IAA ( $\kappa=0.309$ ), reflecting the inherent ambiguity of intent in short texts.
- Schema B (Sahitaj et al., 2025): A span-based, technique-centric taxonomy adapted for Twitter. For this study, the authors clustered its 17 techniques into six high-level categories. This schema demonstrated substantially higher IAA ( $\kappa=0.594$ ).
Models:
Four language models were evaluated: GPT-4.1-nano, Phi-4 14B, Qwen2.5-14B, and Qwen3-14B. Experiments were conducted in both zero-shot (Base) and fine-tuned (FT) regimes. Fine-tuning was performed using LoRA adapters for open-weight models and API-based optimization for GPT-4.1-nano.
Prompting Strategies:
- Direct-High: Models predict the high-level category directly from the text.
- HiPP (Hierarchical Prompting): A "Main→High" strategy where the model first predicts the most prominent fine-grained technique and then aggregates this prediction to determine the high-level category in a single pass.
Evaluation:
Performance was measured using Macro-F1 and Weighted-F1 on a balanced golden test set (200 tweets per schema). Error analysis included confusion matrices, lemma-frequency statistics, and PCA visualization of TF-IDF vectors.

Key Contributions

New Intent-Based Taxonomy: The paper introduces the first entirely new conceptual taxonomy for propaganda since 2019, shifting focus from surface-level techniques to strategic communicative goals.
Controlled Comparative Framework: The study isolates the effects of schema design, supervision, and prompting by treating them as orthogonal factors, demonstrating how these elements interact to shape model robustness.
Hierarchical Prompting (HiPP) Validation: The authors propose and validate the HiPP method, showing its specific utility in fine-tuned settings, particularly for noisy, low-agreement annotation schemas.
High-Level Labeling for Existing Schemas: The authors created high-level labels for the Sahitaj et al. (2025) schema via a clustering procedure, achieving high agreement in human validation and providing a robust baseline for comparison.

Results

Impact of Fine-Tuning: Fine-tuning was decisive across all configurations, transforming weak zero-shot baselines into competitive systems (improvements of +0.09 to +0.30 in Weighted-F1). It revealed methodological differences that were invisible in the base regime.
Schema Performance: The Sahitaj schema consistently yielded higher absolute performance (gaps of +0.05–+0.15 in Weighted-F1) due to its higher annotation reliability. However, the intent-based schema, while more challenging, provided a richer lens on strategic goals.
Model Performance: Among open-weight models, the Qwen series (specifically Qwen3-14B) achieved the strongest overall performance. Phi-4 14B consistently outperformed GPT-4.1-nano across both schemas and regimes.
Effectiveness of HiPP:
- In zero-shot settings, HiPP (Main→High) often underperformed Direct-High, particularly on the low-IAA schema, suggesting hierarchical decomposition can amplify noise without adapted representations.
- After fine-tuning, HiPP became highly beneficial, especially on the intent-based schema. It provided a stable methodological advantage, helping models recover majority-vote labels even in cases of high annotator disagreement.
Error Analysis:
- Zero-Shot: Models were hyper-sensitive to lexical cues (e.g., "Ukraine," "Nazi," "war"), leading to high false positives.
- Fine-Tuned: Errors shifted from simple lexical triggers to genuinely ambiguous, polarized cases. Fine-tuned HiPP models reduced false positives on neutral war reporting and concentrated errors on conceptually neighboring categories (e.g., confusing Distort Reality with Shift Blame), indicating a more structured understanding of the taxonomy.
- PCA Visualization: The intent-based schema produced a denser, overlapping decision space compared to the compact, well-separated clusters of the Sahitaj schema, confirming the higher difficulty of the intent-based task.

Significance and Claims
The paper claims that its primary contribution is methodological, demonstrating that fine-tuning is essential for robust propaganda detection and that hierarchical prompting (HiPP) acts as a stabilizing inductive bias specifically when models are adapted to noisy, imbalanced annotation spaces.

The authors argue that while simplified, technique-centric schemas offer clearer learning signals, intent-based schemas are necessary for nuanced analysis of propaganda's strategic goals. They posit that modern LLMs, when fine-tuned with HiPP, converge on similar, interpretable error structures, suggesting that conclusions drawn from one model family are reasonably transferable to others. The work provides a challenging benchmark (the HQP dataset with new intent-based labels) for future research on real-world detection, emphasizing that robust systems must account for the trade-off between ecological realism and annotation reliability.

The authors remain modest, noting that their results are indicative of methodological trends based on a single data split and that the proposed intent-based taxonomy, while valuable, introduces inherent noise that makes it a more difficult classification problem than existing frameworks.