Reliable News or Propagandist News? A Neurosymbolic Model Using Genre, Topic, and Persuasion Techniques to Improve Robustness in Classification

1. Problem Statement

The paper addresses the challenge of detecting propagandist news, a specific form of "news disorder" where oriented messages are smuggled within factual reports to mimic reliable journalism.

• Limitations of Current Approaches: Existing methods, particularly those based on Large Language Models (LLMs) like BERT or RoBERTa, achieve high accuracy on specific datasets but suffer from overfitting. They often rely on dataset-specific artifacts (e.g., specific source domains or vocabulary) rather than learning generalizable features, leading to poor performance when applied to new sources, political contexts, or credibility levels.
• The Goal: To develop a classification model that is robust and generalizes well to unseen news sources and contexts, moving beyond pure text-based pattern matching.

2. Methodology: The Neurosymbolic Approach

The authors propose a neurosymbolic hybrid model that combines non-contextual neural text embeddings with symbolic, interpretable conceptual features.

A. Datasets

The study utilizes two corpora introduced in prior work (Faye et al., 2024):

1. PPN (Propagandist Pseudo-News): 12,427 articles from outlets identified as propaganda sources (e.g., by NewsGuard and VIGINUM) covering the Russia-Ukraine conflict in 9 languages.
2. MAINSTREAM: 2,371 articles from established newspapers (English and French) serving as the control group.

• Focus: The experiments focus on the English subset (3,219 PPN + 1,004 MAINSTREAM) due to better model performance on this language.

B. Feature Extraction

The model integrates three layers of information:

1. Neural Component (Text Embeddings):
   • Uses fastText (non-contextual) to generate a 300-dimensional vector representing the text's distributional lexical characteristics.
2. Symbolic Component (Conceptual Features):
   • Genre: One-hot encoded vector (3 dimensions: Reporting, Opinion, Satire).
   • Topic: One-hot encoded vector (9 dimensions based on topic distribution).
   • Persuasion Techniques: A vector counting the frequency of specific techniques.
     • Fine-grained: 23 dimensions (specific techniques).
     • Coarse-grained: 6 dimensions (grouped techniques).
   • Total Symbolic Vector: 35 dimensions (3 + 9 + 23).

C. Model Architecture

• Input: Concatenation of the 300-dim fastText vector and the 35-dim symbolic vector (Total: 335 dimensions).
• Network: A simple two-layer perceptron (MLP):
  1. Dense layer (335 $\to$ 335) with ReLU activation.
  2. Dense layer (335 $\to$ 1) with Sigmoid activation.
• Output: A probability score (0–1) for propaganda likelihood.
• Rationale: The architecture is intentionally kept simple to facilitate explainability, which is difficult with complex LLMs.

D. Evaluation Strategy (Robustness Testing)

To test generalization, the authors designed four specific data splits (Train/Valid/Test) rather than standard random splits:

1. Random: Standard 80/10/10 random split.
2. Sources: Training on specific sources; testing on entirely different sources.
3. Political: Training on specific political leanings (e.g., Left); testing on unseen leanings (e.g., Right).
4. Credibility: Training on high-credibility sources; testing on low-credibility sources (and vice versa for propaganda).

3. Key Contributions

1. Neurosymbolic Framework: Demonstrates that combining static text embeddings with explicit, human-interpretable features (genre, topic, persuasion) significantly improves robustness against domain shifts.
2. Bias-Aware Evaluation: Introduces a rigorous evaluation protocol using "biased splits" (Source, Political, Credibility) to explicitly measure a model's ability to generalize to new contexts, rather than just memorizing dataset artifacts.
3. Explainability: By using a simple architecture and SHAP (SHapley Additive exPlanations) values, the authors provide insights into why the model makes decisions, showing that symbolic features become critical when text embeddings are insufficient.

4. Results

The study compares three models: Text Only (fastText), Hybrid Lite (Coarse-grained persuasion), and Hybrid (Fine-grained persuasion).

• Performance on Random Splits:
  • All models performed similarly well (Accuracy ~79%, F1 ~88%). The "Text Only" model performed slightly better here, suggesting text features are sufficient when training and test distributions align.
• Performance on Robustness Splits (The Critical Finding):
  • Sources & Credibility Splits: The Text Only model collapsed completely (F1 score dropped to 0.0 in some cases), failing to distinguish propaganda from reliable news when sources changed.
  • Hybrid Method: Maintained high performance (F1 scores between 81% and 88%).
  • Hybrid Lite: Performed well on Credibility but struggled significantly on Source splits (F1 ~31%), indicating that fine-grained persuasion techniques are crucial for source generalization.
• Ablation Analysis:
  • The Hybrid method showed an average improvement of +26.89% in Accuracy and +41.85% in F1-score over Text Only across the robustness splits.
  • The Hybrid method had the lowest variance, confirming its stability.
• Explainability (SHAP Analysis):
  • In Random splits, text embeddings were the primary drivers of the decision.
  • In Credibility and Source splits (where text embeddings failed), persuasion technique embeddings became the dominant feature, proving the model successfully relied on symbolic cues to generalize.

5. Significance and Conclusion

• Robustness over Accuracy: The paper argues that high accuracy on random splits is misleading for propaganda detection. True robustness requires models that can handle shifts in source, political bias, and credibility.
• The Role of Symbolic Features: The study proves that explicit features like persuasion techniques and genre are not just auxiliary but essential for detecting propaganda when the text itself is ambiguous or comes from an unseen domain.
• Practical Application: The proposed model offers a lightweight, explainable alternative to heavy LLMs, suitable for real-time detection where understanding why an article is flagged is as important as the flag itself.
• Future Work: The authors suggest extending the corpus to other themes (elections, other conflicts) and languages, and potentially integrating external source reliability scores directly into the model.

In summary, this paper demonstrates that hybrid neurosymbolic models are superior to pure deep learning approaches for detecting propaganda in dynamic, real-world scenarios, primarily because they leverage structural and rhetorical cues that remain consistent even when the source or specific vocabulary changes.