Let the Bullets Fly: Multimodal Fake News Detection with Temporal-Aligned Generative Danmaku

Technical Summary: Let the Bullets Fly

Problem Definition
The proliferation of short-video platforms has accelerated the spread of fake news, necessitating real-time detection strategies. While social interaction signals (e.g., user comments) provide valuable discriminative cues, their application to "Danmaku" (bullet comments overlaying video playback) in fake news detection remains underexplored. The primary obstacle is the inherent accumulation latency of real-world Danmaku; genuine user interactions often lag behind the video, violating the real-time requirements of detection systems. Furthermore, existing methods struggle to capture fine-grained, temporal inconsistencies within video streams, often relying on global content analysis or static social signals that miss fleeting segments of fabricated information.

Methodology
The authors propose a two-stage framework to bridge the gap between user behavior and real-time detection: a temporal generative model (Genda) and a detection model (DM-FEND).

1. Genda: Temporal Generative Danmaku Framework
   Genda simulates the time-aware interactive process of crowd responses to reconstruct realistic pseudo-Danmaku streams without waiting for real accumulation. It consists of two collaborative modules:

   • Video Understanding: The input video is decomposed into short, non-overlapping segments (4 seconds each). A vision-language model (VLM) extracts segment-level representations, encoding semantic content, emotional tone, and relative temporal position.
   • Danmaku Trigger: This module models the timing and intensity of user reactions. Instead of simple classification, it treats reaction intensity as an ordinal latent variable (ranging from "None" to "Heated"). It uses a Large Language Model (LLM) with LoRA adaptation to predict the probability of reaction levels based on semantic, emotional, and contextual signals.
   • Danmaku Generator: Conditioned on the video segment representation and the predicted reaction intensity, this module synthesizes semantically coherent and emotionally diverse bullet comments. It incorporates a latent style variable to simulate diverse user personas, ensuring the generated stream is temporally aligned with the video frames.

2. DM-FEND: Danmaku-guided Temporal Multimodal Detection
   DM-FEND utilizes the generated Danmaku as intermediate supervisory signals to enhance fake news detection across video, audio, and text modalities.

   • Danmaku-guided Unimodal Learning: The model employs a masking and reconstruction mechanism. It computes saliency scores for tokens in video and audio modalities based on their alignment with the generated Danmaku. High-saliency tokens (relevant to user reactions) are reconstructed to reinforce semantic cues, while low-saliency tokens are masked to suppress semantic noise.
   • Danmaku-guided Multimodal Learning: The framework models pairwise interactions between modalities (text-video, text-audio, video-audio). It uses contrastive learning to align these modality pairs with the Danmaku signal, encouraging consistency for matching segments and pushing apart mismatched ones.
   • Classification: The model aggregates the temporally aligned multimodal representations and feeds them into a classifier to determine the authenticity of the video.

Key Contributions

• Introduction of Danmaku to Fake News Detection: The study is the first to integrate Danmaku into multimodal fake news detection, addressing the latency issue by proposing Genda, a framework that generates temporally aligned, human-like pseudo-Danmaku.
• Novel Detection Architecture: The authors developed DM-FEND, which leverages generated Danmaku to facilitate fine-grained interactions among video, audio, text, and social signals. This design enhances dynamic modality alignment and suppresses semantic noise through a dual masking mechanism.
• Empirical Validation: Extensive experiments on two benchmarks (Chinese FakeSV and English FakeTT) demonstrate the efficacy of the approach.

Experimental Results
The proposed DM-FEND consistently outperforms state-of-the-art baselines, including unimodal models, LLM prompting benchmarks, and existing multimodal detectors.

• Performance: On the FakeSV dataset, DM-FEND achieved 87.01% Accuracy and 86.68% Macro-F1, surpassing the best baseline (FakeSV-VLM) by 0.85% and 0.94%, respectively. Notably, it showed a significant improvement in Recall (90.66%), indicating a stronger ability to detect difficult or ambiguous fake news.
• Cross-Dataset Generalization: On the FakeTT dataset, DM-FEND improved accuracy by 3.63% and Macro-F1 by 3.73% over the best baseline (ExMRD).
• Ablation Studies: Removing any component (Temporal Danmaku Generation, Unimodal Learning, or Multimodal Learning) resulted in performance degradation. The full model showed the most distinct separation between real and fake samples in feature space (visualized via PCA), confirming the complementary nature of the modules.

Significance and Claims
The paper claims to offer a "bright and robust solution" for multimodal fake news detection in modern social interactive contexts. By simulating the temporal evolution of user interactions, the study bridges the inconsistency between news content and user behavior, allowing detectors to identify fleeting segments of fabricated information that global analysis might miss. The authors emphasize that their approach does not merely rely on content authenticity but explicitly models how users perceive and react to content over time, thereby capturing local cross-modal inconsistencies and improving robustness against sophisticated manipulations. The code and data are released to ensure reproducibility.