Multi-Agentic System Leveraging Open-Source LLMs to Mitigate Disinformation Threats

Technical Summary: Multi-Agentic System Leveraging Open-Source LLMs to Mitigate Disinformation Threats

Problem Statement
The proliferation of disinformation, accelerated by social media and artificial intelligence, has rendered manual fact-checking inadequate due to the sheer scale and velocity of information dissemination. While automated detection is necessary, reliance on single Large Language Models (LLMs) often lacks the nuance of human collaborative decision-making. Furthermore, many existing solutions rely on proprietary, closed models (e.g., GPT-4), which limits transparency and reproducibility. There is a critical need for automated systems that not only detect disinformation but also emulate the consensus-building and diverse knowledge bases of human annotation teams, particularly across languages with varying resource availability (high, medium, and low).

Methodology
The authors propose a novel Multi-Agent System (MAS) designed to emulate the cognitive decision-making processes of human annotators. The system operates on a "hub-and-spoke" topology consisting of four agents:

1. Leader Agent: Functions as a central moderator responsible for task assignment, initiating debates, and synthesizing final decisions. The leader can adjudicate if consensus is unattainable.
2. Three Expert Annotator Agents: Operate as equals, providing initial annotations and justifications.

Key Architectural Features:

• Heterogeneous LLM Integration: Unlike systems using a uniform model for all agents, this MAS utilizes diverse open-source models (e.g., LLaMA-3.1-405B, Kimi, Deepseek, Qwen, LLaMA-Nemotron) to simulate the natural diversity in knowledge and cognitive abilities found in human teams.
• Consensus-Driven Debate: The system mimics human workflows where decisions are reached through collective conviction.
  • Scenario 1 (Immediate Consensus): If all agents agree, the leader formats the result immediately.
  • Scenario 2 (Debate): If opinions diverge, the leader moderates an iterative debate (up to 20 rounds) where agents revise their justifications and decisions based on peer input.
  • Scenario 3 (Adjudication): If consensus fails, the leader makes an autonomous final decision based on the accumulated arguments.
• Closed Configuration: The system relies strictly on the internal knowledge and reasoning capabilities of the LLMs, deliberately excluding external internet search tools. This is designed to predict disinformation before professional fact-checkers have debunked it.
• Transparency: The entire debate transcript is archived, allowing for human observation and analysis of the reasoning process.

Key Contributions
The paper identifies five primary contributions:

1. Comprehensive MAS Architecture: A system capable of addressing diverse disinformation tasks, including direct detection, check-worthy claim identification, and verifiable factual claim detection.
2. Generalizability: The system demonstrates robust performance across high-resource (English), medium-resource (Polish), and low-resource (Slovak, Bulgarian) languages without task-specific fine-tuning.
3. Human-Inspired Consensus: The design grounds decision-making in the principle of consensus, mirroring how human annotators strengthen accuracy through collective agreement.
4. Heterogeneous Knowledge Representation: The use of diverse open-source models ensures that justifications are grounded in varied knowledge bases and reasoning methodologies.
5. Interpretability: The provision of debate transcripts offers a novel reporting mechanism for analyzing agent deliberation.

Experimental Results
The system was evaluated on four tasks across four languages using benchmark datasets (MIPD, CLEF23, CLEF22, and a self-collected Slovak dataset):

• Task 1 (Polish Disinformation Detection): The MAS achieved 0.88 accuracy and a 0.87 macro F1-score. Notably, it achieved a recall of 0.98 for disinformation instances, outperforming standalone GPT-4 and GPT-3.5, though it slightly trailed a self-conducted GPT-5.2 baseline.
• Task 2 (English Check-Worthy Claims): The system achieved 0.80 accuracy and a 0.79 macro F1-score. It demonstrated superior recall (0.89) for check-worthy claims compared to single-model baselines, indicating high sensitivity in identifying content requiring verification.
• Task 3 (Slovak Check-Worthy Claims): On a small, unseen dataset, the MAS achieved a recall of 0.967, validating its predictive capability in low-resource scenarios.
• Task 4 (Bulgarian Verifiable Factual Claims): The system achieved 0.80 accuracy and a 0.79 macro F1-score, outperforming single LLM configurations (including LLaMA-3.1-405B and GPT-5.2) on the same dataset.

Significance and Claims
The authors claim that this work offers a valuable alternative to fine-tuned models and proprietary closed systems. While the system does not claim to achieve state-of-the-art performance across all metrics compared to heavily fine-tuned models, it highlights significant strengths:

• Robustness in Small Data: The system performs effectively without the need for large, task-specific training datasets, avoiding the costs and biases associated with fine-tuning.
• Transparency and Ethics: By leveraging open-source models and providing debate logs, the system enhances reproducibility and ethical accountability.
• Human-Level Mimicry: The qualitative analysis of debate transcripts confirms that the system successfully models human consensus mechanisms and knowledge diversity.
• Practical Utility: The high recall rates suggest the system is particularly effective as a screening tool to identify texts worthy of human fact-checking, potentially reducing the workload on human annotators while minimizing the risk of overlooking disinformation.

The paper concludes that while the MAS is not a perfect replacement for all fine-tuned solutions, its ability to generalize across languages, operate without external data retrieval, and provide transparent reasoning makes it a robust and practical tool for the fight against disinformation.