💬 NLP

Avoiding Over-smoothing in Social Media Rumor Detection with Pre-trained Propagation Tree Transformer

To address the over-smoothing and long-range dependency limitations of Graph Neural Networks in social media rumor detection, this paper proposes P2T3, a pre-trained Transformer-based method that models conversation chains within propagation trees and achieves state-of-the-art performance across multiple benchmarks and few-shot scenarios.

Original authors: Chaoqun Cui, Caiyan Jia

Published 2026-03-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chaoqun Cui, Caiyan Jia

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Echo Chamber" Effect

Imagine you are trying to figure out if a piece of news is a rumor or the truth. You look at a social media post and all the replies.

Most computer programs used to do this job (called Graph Neural Networks or GNNs) work like a game of "Telephone." They pass information from one person to their neighbors, then to their neighbors' neighbors, layer by layer.

The Problem: In social media, most people reply directly to the original post. They don't reply to each other. This creates a "star" shape: one center (the original post) with hundreds of spokes (replies) sticking out.

When the "Telephone" game tries to play on this star shape, something weird happens. Because everyone is so close to the center, the information gets mixed up so quickly that everyone ends up sounding exactly the same.

The Metaphor: Imagine a room where everyone is shouting at the same time. If you stand in the middle, you can't tell who is saying what. The unique voices (the specific details that prove a rumor is fake) get drowned out. In computer science, this is called "Over-smoothing." The model becomes "smooth" but useless because it can't distinguish between a true story and a fake one anymore.

The Old Solution vs. The New Idea

The Old Way (GNNs):
The old models tried to fix this by building deeper layers (more rounds of the Telephone game). But on social media, this just made the "smoothing" worse. It's like trying to hear a whisper in a hurricane by turning up the volume on the wind.

The New Way (P2T3):
The authors, Cui and Jia, realized that social media conversations aren't really a messy web; they are actually a series of straight lines.

The Metaphor: Instead of looking at the whole messy room, imagine taking every single conversation thread (the original post + Reply A + Reply to A + Reply to that) and laying them out in a straight line, like a train track.

They built a new model called P2T3 (Pre-Trained Propagation Tree Transformer). Instead of using the "Telephone" game, P2T3 uses a Transformer (the same technology behind Chatbots like me).

How P2T3 Works (The Magic Tricks)

Unraveling the Tree:
The model takes the messy "star" shape of the social media post and pulls it apart into individual conversation chains. It treats each chain like a sentence in a book.
The "Name Tags" (Token-wise Embeddings):
Since the model is now reading a list of sentences, it needs to know which sentence belongs to which chain and how deep it goes.
- Chain ID: It gives every conversation a unique "name tag" so it knows, "This reply belongs to the Trump thread, not the Brown thread."
- Depth: It marks how far down the reply is (1st reply, 2nd reply, etc.).
- Type: It labels if it's the original post or a deep conversation.
- Why this matters: This stops the model from getting confused. It knows exactly who is talking to whom, preventing the "Over-smoothing" mess.
The "Super-Reading" (Pre-training):
Before the model tries to detect rumors, it reads hundreds of thousands of unlabeled social media posts.
- The Analogy: Imagine a detective who reads every newspaper in the world for a year before they start solving a specific crime. They learn how people argue, how emotions flare up, and how rumors spread naturally.
- P2T3 does this by learning to connect the original post with its first replies. It learns the "vibe" of a rumor without needing a teacher to tell it "this is fake" every single time.

Why It's a Game Changer

It Doesn't Get Confused: Because it reads the conversation chains in order (like a story), it doesn't suffer from the "everyone sounds the same" problem. It can hear the deep, angry, or emotional replies at the bottom of the thread that often give away a rumor.
It Learns Fast: Because it was pre-trained on massive amounts of data, it works incredibly well even when you only give it a few examples (Few-Shot learning). This is crucial because rumors often get deleted before we can collect enough data to train a model.
It Scales Up: You can make the model bigger and smarter without it breaking, which is impossible for the old "Telephone" models.

Summary in One Sentence

The authors realized that old rumor detectors were getting "dizzy" trying to process social media trees, so they invented a new system that untangles the mess into straight conversation lines, gives them unique ID tags, and lets a powerful AI read them like a storybook to spot lies much faster and more accurately.

1. Problem Statement

The paper addresses a critical limitation in current deep learning approaches for social media rumor detection: over-smoothing when using Graph Neural Networks (GNNs) to analyze Rumor Propagation Trees (RPTs).

The Phenomenon: Existing methods typically model rumor propagation as a tree structure and use GNNs to learn node representations. However, the authors observe that as GNN depth increases, performance often degrades rather than improves.
The Root Cause: The authors identify that RPTs have a unique, highly imbalanced structural characteristic:
- Shallow Depth: The vast majority of nodes (comments) are 1-level nodes (direct replies to the source post).
- Indiscriminate Aggregation: In GNNs, the receptive field expands with depth. Because most nodes are directly connected to the root, even shallow GNNs allow 1-level nodes to aggregate information from all other 1-level nodes within 2 hops. This leads to "over-smoothing," where node features converge to indistinguishable values.
- Loss of Long-Range Dependencies: GNNs struggle to capture long-range dependencies within complete conversation chains (from a 1-level node down to a leaf node), which often contain crucial semantic and emotional cues for detecting rumors.
The Contradiction: While bidirectional information (top-down and bottom-up) is theoretically beneficial, using undirected graphs in GNNs to capture this often leads to performance degradation due to the over-smoothing inherent in the RPT structure.

2. Methodology: P2T3 (Pre-Trained Propagation Tree Transformer)

To circumvent the limitations of GNNs, the authors propose P2T3, a method based on a pure Transformer architecture designed specifically for RPTs.

A. Core Architecture Shift

Instead of treating the RPT as a graph for message passing, P2T3 transforms the tree into sequential conversation chains.

Extraction: The model extracts all conversation threads from the root to the leaf nodes, following the propagation direction (source $\to$ reply $\to$ reply).
Sequential Modeling: These chains are treated as sequential structures, allowing the Transformer's self-attention mechanism to capture long-range dependencies and global context without the risk of over-smoothing.

B. Input Representation & Token-wise Embeddings

To enable the Transformer to understand the tree structure without explicit graph edges, P2T3 introduces three specific token-wise embeddings to augment the input sequence:

Chain Identifier: Orthogonal vectors assigned to tokens belonging to the same conversation chain. This ensures the model distinguishes between different branches of the tree.
Depth Embedding: Sinusoidal positional encodings representing the depth of a node in the original tree. This helps the model understand the hierarchical reply relationship.
Type Embedding: Vectors indicating the conversation type: Source (root), Deep Conversation, or Shallow Conversation. This provides inductive bias regarding the nature of the interaction.

C. Training Strategy

P2T3 employs a two-stage training process to leverage large-scale unlabeled data:

Pre-training (Unsupervised):
- Dataset: Two massive unlabeled datasets were constructed: UWeibo (209k claims) and UTwitter (204k claims).
- Objective: Maximize Mutual Information (MI) between the source post representation and the first token of all conversation chains (1-level nodes). This aligns global (source) and local (reply) representations.
Fine-tuning (Supervised):
- The model is fine-tuned on labeled rumor detection datasets using standard cross-entropy loss.
- The unsupervised loss can be weighted and added during fine-tuning to further enhance performance.

3. Key Contributions

Empirical & Theoretical Analysis of Over-smoothing: The paper provides extensive evidence that RPTs are inherently prone to over-smoothing in GNNs due to their shallow, imbalanced structure (high density of 1-level nodes). It demonstrates that increasing GNN depth often harms performance.
P2T3 Model: A novel Transformer-based architecture that converts RPTs into sequential conversation chains, utilizing specific token-wise embeddings to preserve structural information while avoiding over-smoothing.
Large-Scale Unlabeled Datasets: The release of UWeibo and UTwitter, two large-scale unlabeled datasets containing over 200,000 claims each, to facilitate semi-supervised rumor detection research.
State-of-the-Art Performance: The model achieves superior results across multiple benchmarks and demonstrates robustness in few-shot scenarios.

4. Experimental Results

The authors evaluated P2T3 on four benchmark datasets: Weibo, DRWeibo, Twitter15, and Twitter16.

Performance vs. SOTA: P2T3 outperformed all baseline methods (including GNN-based models like BiGCN, RAGCL, DDGCN, and Transformer-based models like PLAN and HD-TRANS) in Accuracy, Precision, Recall, and F1 scores.
- Example: On Weibo, P2T3 achieved 97.3% Accuracy, surpassing the previous best (RAGCL at 96.0%).
Ablation Studies:
- Removing Token-wise Embeddings caused a significant drop in performance (e.g., -5.2% on Weibo), proving the necessity of encoding chain and depth information.
- Pre-training on large unlabeled data provided a consistent boost (+1.5% on Weibo) compared to training from scratch.
Scalability (Model Depth): Unlike GNNs, which suffer performance degradation as layers increase, P2T3's performance improves and stabilizes as the number of Transformer layers increases, confirming it is free from over-smoothing.
Few-Shot Learning: P2T3 demonstrated strong performance with limited labeled data (10–500 samples), leveraging its pre-trained representations to generalize effectively.

5. Significance and Future Implications

Paradigm Shift: The paper challenges the dominance of GNNs in rumor detection, arguing that for the specific topology of social media propagation trees, Transformer architectures are superior.
Scalability: By avoiding over-smoothing, P2T3 enables the use of deeper, larger models and the effective utilization of massive unlabeled social media data, which was previously difficult with GNNs.
Future Directions: The authors suggest that P2T3 serves as a foundation for future unified multi-modal schemes (combining text, images, and graph structures) and can be extended to other social media tasks like content recommendation and user behavior analysis. It also paves the way for integrating Large Language Models (LLMs) into rumor detection frameworks.