Conspiracy Frame: a Semiotically-Driven Approach for Conspiracy Theories Detection

1. Problem Statement

Conspiracy theories (CTs) are anti-authoritarian narratives that foster social discord and distort the perception of political information. While Natural Language Processing (NLP) has made strides in detecting misinformation, current approaches face two critical limitations:

• Computational Limitation: Existing datasets typically rely on binary or multi-label classification (e.g., "conspiracy" vs. "non-conspiracy"). This approach lacks granularity, failing to distinguish CTs from other information disorders (like propaganda or disinformation) because it ignores the internal structural and semantic architecture of the narratives.
• Thematic Limitation: Most existing resources are confined to specific topics (e.g., COVID-19, vaccines), limiting the generalizability of detection models to new domains or evolving conspiracy themes.

The authors argue that effective detection requires moving beyond surface-level topic classification to a fine-grained semantic representation based on the structural elements inherent to conspiratorial discourse.

2. Methodology

The paper proposes a novel framework combining semiotics and frame semantics to define and detect conspiracy theories.

A. Theoretical Framework: The Conspiracy Frame

Drawing from semiotic literature (Birchall, 2006; Leone et al., 2021) and Frame Semantics (Fillmore, 1976), the authors define a "Conspiracy Frame" consisting of five specific elements that must be present (at least the core ones) for a narrative to be classified as a conspiracy theory:

1. Plan (Core): The event or action taken by a malicious group to achieve a harmful end (e.g., "forced vaccination," "ethnic replacement").
2. Secret (Core): The element of secrecy or hidden coordination surrounding the plan (e.g., "silencing the truth," "Big Media misleading").
3. In-group (Non-core): The victims ("us") harmed by the plan.
4. Out-group (Non-core): The enemy ("them") executing the plan.
5. Call-to-Action (Non-core): The imperative to prevent a catastrophic outcome.

B. Dataset Creation: Con.Fra.

The authors constructed the Conspiracy Frames (Con.Fra.) dataset through a three-step iterative process:

1. Data Collection: Massive crawling of Telegram channels (a known hub for CTs), resulting in 11,618 channels. They selected the top 35 channels by subscriber count to ensure diversity, collecting 2 years of messages (2023–2024).
2. Annotation:
   • Phase 1 (Corpus Creation): 13 annotators (PhD/MA students) annotated 2,077 messages. They identified CTs and marked spans corresponding to the five frame elements.
   • Phase 2 (LLM Evaluation): An additional 500 messages from unseen channels were used to evaluate LLM generalization.
   • Reliability: Inter-annotator agreement was moderate for binary classification (Kappa = 0.41) but substantial for span detection (Kappa ranging from 0.63 to 0.80), with the "Plan" element showing the highest agreement.
3. Lexical Mapping: Annotated spans were lemmatized and mapped to FrameNet (v1.7) to identify abstract semantic patterns (e.g., mapping "genocide" to the Killing frame).

C. Experimental Setup

The authors evaluated Large Language Models (LLMs), specifically LLaMA-3.3 (8B and 70B parameters), using three in-context learning strategies:

1. Zero-Shot: Basic definition of CTs.
2. Few-Shot: Definition + annotated examples.
3. Frame-Guided: Definition + examples + explicit FrameNet hints (e.g., suggesting specific frames like EXECUTE_PLAN or SECRECY_STATUS to guide the model).

The evaluation included:

• In-domain: Testing on the annotated Con.Fra. corpus.
• Out-of-domain: Testing on 500 messages from unseen Telegram channels.
• Metrics: Precision, Recall, F-score, and an ELO rating system for pairwise human evaluation of model outputs.

3. Key Contributions

1. The Conspiracy Frame: A semiotically-driven, fine-grained semantic representation of CTs that moves beyond topic-based classification to structural narrative analysis.
2. Con.Fra. Dataset: The first span-level annotated corpus of Telegram messages specifically designed for CT detection, covering diverse geopolitical topics and annotated with the five frame elements.
3. FrameNet Alignment: A methodology for mapping conspiratorial spans to abstract FrameNet frames, revealing underlying semantic patterns (e.g., Kinship, Ingest_substance, Political_Locales) that characterize CT rhetoric.
4. Comprehensive Evaluation: A rigorous assessment of LLM capabilities in both in-domain and out-of-domain settings, including a novel use of ELO ratings to compare prompting strategies.

4. Results

A. Model Performance (In-Domain)

• Model Size: The 70B model outperformed the 8B model by ~3.1 F-score points, indicating that larger models handle the nuance of CTs better.
• Prompting Strategies:
  • Classification: Zero-shot, Few-shot, and Frame-guided approaches achieved substantially equal F-scores.
  • Trade-off: Zero-shot excelled in Precision (+9 points over others), while Few-shot and Frame-guided significantly improved Recall (+14 to +15 points).
  • Span Detection: Zero-shot performed poorly across all elements. Frame-guided approaches showed slight improvements in detecting In-group and Call-to-action, while Few-shot was slightly better for Out-group.
• Key Finding: Injecting FrameNet knowledge into prompts did not lead to a clear, consistent increase in performance over standard few-shot prompting.

B. Out-of-Domain Evaluation (ELO Ratings)

• Size Effect: The 70B model consistently outperformed the 8B model in both classification and span detection (winning ~99% of ELO comparisons).
• Frame vs. Few-Shot: Results were mixed. The Frame-guided approach won in text classification (830/1000 wins) but lost significantly in span detection (213/1000 wins) against the Few-shot approach.
• Qualitative Error Analysis:
  • Frame-guided strengths: Better at capturing full narrative scope, complex mechanisms, and implicit "secrets."
  • Frame-guided weaknesses: Tendency to over-extract (false positives) or conflate elements (e.g., merging "plan" and "secret") when the text was simple.
  • Few-shot strengths: Better syntactic precision and avoiding unnecessary context in simpler texts.

C. Semantic Pattern Analysis

Mapping spans to FrameNet revealed that CTs rely on specific abstract patterns rather than just specific topics:

• High Recurrence: Frames like Political_Locales (government, state), Statement (telling, reporting), and Ingest_Substance (vaccines, drugs) were frequent.
• Insight: CTs function as counter-narratives where mainstream statements are framed as inherently misleading. The rhetoric often uses metaphors of invasion, contagion, and war.
• Gap: The core CT elements (Plan/Secret) did not map cleanly to the most frequent FrameNet frames, suggesting a need for more specialized semantic modeling.

5. Significance and Future Work

• Explainability: While Frame-guided prompting did not drastically boost automated metrics, the framework provides a structural explanation for why a text is conspiratorial, making detection more interpretable.
• Generalizability: The Con.Fra. dataset and frame-based approach offer a path toward detecting CTs across diverse topics, not just health-related ones.
• Future Directions: The authors propose moving beyond surface-level prompt hints. Future work involves fine-tuning or pre-training LLMs on knowledge graphs constructed from FrameNet frames. This would allow models to internalize frame semantics as structured relational knowledge, potentially enabling deeper reasoning about the structural logic of conspiracy narratives.

Conclusion: The paper demonstrates that while current LLMs can detect CTs, the integration of semiotic and frame-semantic structures is crucial for understanding the mechanism of these narratives. Although direct performance gains from frame-injection were modest in this study, the approach lays the groundwork for more robust, explainable, and generalizable detection systems.