KITE: A Tri-Modal Transformer Integrating Text, Images, and Knowledge Graphs for Fake News Detection

Technical Summary: KITE – A Tri-Modal Transformer for Fake News Detection

Problem Statement
The rapid evolution of digital media has facilitated the spread of sophisticated misinformation that seamlessly blends deceptive text, manipulated visuals, and factually incorrect claims. Traditional detection methods often rely on single-modality analysis (text-only or image-only) or employ bimodal fusion strategies that treat external knowledge as a post-processing step or a separate verification layer. These approaches frequently fail to detect deep semantic inconsistencies, particularly when malicious actors deliberately alter text or images to bypass detection, or when subtle contradictions exist between content and verified real-world facts. Existing frameworks lack the ability to jointly model textual, visual, and factual knowledge representations within a unified core fusion process.

Methodology
To address these limitations, the authors propose KITE (Knowledge-Integrated Text–Image Encoder), a unified tri-modal framework designed to jointly encode textual content, visual cues, and external factual knowledge. The architecture consists of four primary stages:

1. Input Processing:

   • Text Encoding: News article text is processed using RoBERTa to generate rich semantic embeddings that capture contextual meaning.
   • Image Encoding: Visual content is encoded using CLIP (Contrastive Language-Image Pretraining) to align visual features with textual concepts in a shared embedding space.

2. Knowledge Integration:

   • Entity Extraction: A Named Entity Recognition (NER) module (using spaCy) extracts key entities (persons, organizations, locations) from the text.
   • Fact Retrieval: These entities are queried against Wikidata to retrieve relevant factual triples.
   • Graph Encoding: The retrieved entity-fact relationships are constructed into an article-specific mini-knowledge graph. This graph is encoded using a two-layer Graph Attention Network (GAT). Unlike static embedding methods, the GAT dynamically learns pairwise attention weights between nodes, optimizing edge weights specifically for the fake-news classification objective. The output is a 256-dimensional knowledge embedding.

3. Tri-Modal Fusion:

   • The text, image, and knowledge embeddings are treated as distinct tokens and fed into a lightweight Cross-Modal Transformer (CMT).
   • Within this single Transformer layer, a self-attention mechanism allows all three modalities to attend directly to one another. This dynamic alignment enables the model to detect inconsistencies not only between text and images but also contradictions between the content and external factual knowledge.

4. Classification and Interpretability:

   • The fused representation is passed through a linear layer with a sigmoid activation for binary classification (Real/Fake).
   • Simultaneously, the model generates modality-specific confidence scores for text, image, and knowledge streams. These scores are trained jointly with the main classifier under a focal-loss objective, providing interpretability by indicating which input type most influenced the final decision.

Key Contributions
The paper outlines four primary contributions:

• Unified Tri-Modal Fusion: KITE is presented as the first framework where text, image, and knowledge-graph tokens attend directly to one another within a single Transformer layer. This contrasts with prior models (e.g., KAN) that inject knowledge only after a separate bimodal fusion step, allowing KITE to detect semantic contradictions earlier in the processing pipeline.
• Context-Aware GAT Encoding: Instead of using fixed entity embeddings or static Graph Convolutional Networks (GCNs), KITE constructs a dynamic, article-specific knowledge graph. The GAT learns relational weights contextually, providing richer, fact-grounded representations.
• Three-Way Cross-Modality Attention: Inspired by Cross-Modal Augmentation (CMA), KITE's attention mechanism operates across all three modalities simultaneously within the same layer, capturing inconsistencies that bimodal or static-fusion models miss.
• Interpretable Confidence Scoring: The model outputs confidence scores for each modality alongside the final prediction, offering insights into whether the decision was driven by visual cues, textual content, or factual knowledge.

Experimental Results
KITE was evaluated on two benchmark datasets from the FakeNewsNet suite: GossipCop (entertainment news) and PolitiFact (political news). The model was compared against unimodal baselines (BERT, XLNet, CNN) and state-of-the-art multimodal baselines (MCAN, MCAN-A, SpotFake+, QMFND, KAN).

• Performance: KITE achieved state-of-the-art results on both datasets. On GossipCop, it reached an accuracy of 0.885, recall of 0.959, and F1 score of 0.93, outperforming the best multimodal baseline (QMFND) and significantly surpassing knowledge-aware baselines like KAN. On PolitiFact, KITE achieved an accuracy of 0.868 and an F1 score of 0.889, again outperforming all compared models.
• Analysis of Gains: The authors attribute these gains to the tri-modal fusion architecture, which improves recall by catching text-image pairs that conflict with Wikidata, and improves precision by reinforcing authenticity when all three modalities agree. The dynamic GAT encoding and the end-to-end fine-tuning of the final layers of RoBERTa and CLIP also contributed to the performance improvements.

Significance and Limitations
The paper claims that KITE's significance lies in its ability to detect sophisticated fake news where misleading language, photoshopped images, and factual inconsistencies are intertwined. By integrating external knowledge directly into the core fusion process rather than as a post-hoc check, the model offers improved robustness and interpretability.

However, the authors acknowledge specific limitations:

• Failure Modes: The model can struggle with misleading textual content involving sarcasm or satire, and low-quality or irrelevant visual signals can introduce noise.
• Data Constraints: The reliance on the FakeNewsNet dataset required extensive manual filtering to remove text-only articles, potentially constraining generalizability.
• Computational Overhead: The experiments were conducted exclusively on CPU hardware due to resource constraints, leading to significant training times (approx. 20 hours) due to Wikidata retrieval latency.

The paper concludes that while KITE demonstrates superior performance, future work should focus on automatic data augmentation, cross-lingual detection, and deception-aware learning strategies to further enhance efficiency and robustness.