Beyond speculation: Measuring the growing presence of LLM-generated texts in multilingual disinformation

Technical Summary: Beyond Speculation: Measuring the Growing Presence of LLM-Generated Texts in Multilingual Disinformation

Problem Statement
The rapid advancement of Large Language Models (LLMs) and their accessibility via user-friendly interfaces (e.g., ChatGPT) has intensified concerns regarding their misuse in generating disinformation. While research confirms LLMs' capability to produce false content, the actual extent of their deployment in real-world disinformation campaigns remains largely unknown. The scholarly community is divided: some argue that fears are overblown due to natural ecosystem limitations, while others suggest specific "longtail" contexts face overlooked risks. Currently, there is a lack of empirical evidence quantifying the prevalence of machine-generated text within existing, real-world disinformation datasets, particularly across different languages, platforms, and time periods.

Methodology
To address these gaps, the authors conducted a comprehensive empirical analysis using two distinct groups of datasets:

1. Benchmark Datasets: Multilingual labeled datasets (e.g., MULTITuDE, MultiSocial, SemEval, GenAI, MIX) containing ground truth for human-written vs. machine-generated texts. These were used to evaluate the robustness and performance of detection methods on out-of-distribution data.
2. Real-World Disinformation Datasets: Four datasets presumed to contain disinformation, misinformation, or propaganda:
   • MultiClaim: Professionally fact-checked multilingual social media posts (up to March 2024).
   • FakeNews: English news articles regarding US elections (April–October 2023).
   • USC_X: A sampled multilingual US election Twitter/X dataset (May–November 2024).
   • FIGNEWS: Multilingual Facebook posts regarding the Israel-Gaza war (October 2023–January 2024).

Detection Framework:
The study employed two fine-tuned detectors based on the Gemma-2-9b-it model, utilizing QLoRA for efficient fine-tuning:

• Gemma_GenAI: Fine-tuned on the GenAI dataset train split.
• Gemma_MultiDomain: Fine-tuned on the combined train splits of MULTITuDE and MultiSocial.

To ensure robustness, the authors utilized a combined confident detection strategy. A text is classified as machine-generated if one detector predicts the positive class with 100% confidence (probability 1.0) and the other does not predict the negative class with 100% confidence (probability > 0.0). To mitigate false positives in specific languages, the Gemma_GenAI detector's positive predictions were disqualified for Arabic, German, Italian, and Russian if they exceeded a 0.1 false positive rate on benchmark data.

For prevalence estimation, the authors also tracked the Mean Score (the average probability of a text being machine-generated) to observe trends without relying on strict classification thresholds.

Key Results
The study provides the first empirical evidence of LLM-generated content in real-world disinformation contexts, revealing the following:

• Temporal Evolution: There is a clear upward trend in machine-generated content following the public release of ChatGPT (November 2022). In the MultiClaim dataset, the proportion of texts detected as machine-generated rose from 0.93% in 2021 to 1.85% in 2023 (a 99% relative increase). By 2024, the prevalence reached 2.15%. The Mean Score for detectors also increased significantly between 2021 and 2023.
• Language Disparities: Prevalence is not uniform across languages. While English and Spanish have the highest absolute numbers of detected texts, Polish (4.7%) and French (4.2%) show the highest relative proportions of machine-generated content in the MultiClaim dataset. In the FIGNEWS dataset, French texts showed over 10% LLM generation, compared to ~4.7% for English.
• Platform Variations: Differences were observed across social media platforms, with Instagram showing the highest proportion (1.5%), followed by Facebook (1.29%) and Twitter/X (0.64%).
• Dataset-Specific Findings:
  • FakeNews: 2.61% of "real" labeled texts and 3.27% of "fake" labeled texts were identified as machine-generated.
  • FIGNEWS: 3.16% of texts were detected as machine-generated, with higher prevalence in texts labeled "Biased against Israel" (5.11%) compared to "Biased against Palestine" (3.13%).
  • USC_X (2024 Election): Prevalence increased from 0.44% in January 2024 to 2.39% in November 2024, coinciding with the US election period.
• Detection Performance: On benchmark datasets, the combined detection method achieved high precision (ranging from 93.58% to 99.85%), ensuring that identified positive samples are highly likely to be truly machine-generated.

Significance and Contributions
The paper claims three primary contributions to the field:

1. Empirical Validation: It moves beyond speculation by providing the first comprehensive assessment of LLM presence in real-world disinformation datasets, confirming that machine-generated content is already a measurable reality (reaching ~2.15% in 2024).
2. Pattern Recognition: The study reveals that LLM misuse is not uniform but follows targeted patterns across specific languages (e.g., Polish, French) and platforms, suggesting that "longtail" contexts are indeed vulnerable and require specific attention.
3. Methodological Framework: The authors establish a robust analytical framework using fine-tuned detectors and confidence-based aggregation to measure prevalence in out-of-distribution data, offering a replicable approach for future monitoring.

Conclusion
The authors conclude that while LLMs are not yet used for disinformation in volumes significantly exceeding legitimate use, the fears of misuse are justified. The documented increase in machine-generated content, particularly in fact-checked claims and election-related discourse, underscores the urgent need for continued investigation, improved detection methods, and credibility assessment systems to preserve information integrity. The findings suggest that proactive monitoring should focus on high-prevalence contexts, such as specific languages and platforms, to effectively counter AI-driven disinformation.