💬 NLP

TruthStance: An Annotated Dataset of Conversations on Truth Social

This paper introduces TruthStance, a large-scale, human-annotated dataset of Truth Social conversations spanning 2023–2025 that addresses the gap in alt-tech discourse research by providing benchmarks for argument mining and stance detection, alongside LLM-generated labels for extensive pattern analysis.

Original authors: Fathima Ameen, Danielle Brown, Manusha Malgareddy, Amanul Haque

Published 2026-02-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Fathima Ameen, Danielle Brown, Manusha Malgareddy, Amanul Haque

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a giant, bustling town square. For years, researchers have been studying the arguments happening in the main square (Twitter/X) and the big community forums (Reddit). They've built maps of who agrees, who disagrees, and how people try to convince each other.

But there's a newer, smaller, and very specific neighborhood called Truth Social. It's like a private clubhouse where people go because they feel the main square doesn't listen to them. Until now, nobody had a good map of what happens inside this clubhouse.

This paper introduces TruthStance, a massive new map of that neighborhood. Here is the story of how they built it and what they found, explained simply.

1. The Problem: The Missing Map

Researchers wanted to understand how people argue on Truth Social. But the data they had was like looking at a photo of a single person shouting in a crowd. They knew what people said, but they didn't know who they were talking to or how the conversation evolved.

It's like trying to understand a game of chess by only looking at the starting position of the pieces, without seeing the moves that follow. You can't see the strategy, the traps, or the checkmates.

2. The Solution: Building the "Reply Tree"

The authors went out and collected 24,000 conversations (threads) from Truth Social, spanning from 2023 to 2025. They didn't just grab the main posts; they grabbed every single reply, and every reply to those replies.

Think of a conversation as a family tree:

The Root: The original post (the ancestor).
The Branches: The direct replies.
The Leaves: The deep, nested replies that go on for days.

They collected over 523,000 comments and organized them into these trees. This allowed them to see the whole family dynamic, not just the ancestor.

3. The Challenge: Teaching Computers to "Listen"

Now they had the data, but they needed to understand the meaning. They needed to answer two questions for every comment:

Is this an argument? (Is the person making a claim with reasons, or just saying "God bless"?).
What is their stance? (Are they agreeing with the person they replied to, fighting them, or just saying "Hmm"?).

Doing this manually for half a million comments would take a human team a lifetime. So, they used AI (Large Language Models) as their "super-readers."

The Training: They first hired human experts to label a small sample (1,500 conversations) to teach the AI what "agreement" and "disagreement" look like.
The Test: They tried different "prompting" techniques (giving the AI different instructions) to see which one worked best. It was like trying different teaching styles to see which student learns the fastest.
The Winner: They found that a specific AI model, when given a mix of examples and step-by-step reasoning instructions, became the best "listener." They used this AI to label the rest of the dataset.

4. What They Discovered: The "Town Square" Dynamics

Once the map was complete, they looked for patterns. Here are the big findings:

Toxicity = Arguments: They found that posts that were longer, more negative, and "toxic" (mean or aggressive) were much more likely to be actual arguments. Short, polite, or prayer-like posts were usually just statements, not debates.
- Analogy: It's like a noisy bar fight. The louder and more aggressive the shouting, the more likely it is a real argument. The quiet whispers are usually just people saying hello.
The "Drift" Effect: As a conversation gets deeper (going further down the reply tree), people stop arguing directly with the original poster.
- Analogy: Imagine a debate starting at the front of a room. The first few people shout at the speaker. But as you move to the back of the room, the people aren't shouting at the speaker anymore; they are just chatting with each other, or nodding politely. They've lost interest in the original fight.
The "Neutral" Wall: If someone replies with a neutral comment (no opinion), the conversation often hits a dead end. The AI couldn't figure out what the next person meant because the "chain of logic" was broken.
- Analogy: If you ask a question and someone says, "Maybe," the conversation usually stops there. You don't know if the next person is agreeing with "Maybe" or the original question.

5. Why This Matters

This paper is like handing researchers a microscope for a specific type of online culture.

For Scientists: It helps them understand how political ideas spread in "alternative" spaces that are different from mainstream Twitter.
For the Public: It shows us that even in a place designed for one side of the political spectrum, people still argue, still drift away from the main topic, and still get tired of fighting.

The Bottom Line

The authors didn't just dump a pile of data on the internet. They built a structured, annotated library of conversations, taught an AI to read it, and used it to reveal how people argue in a specific, under-studied corner of the internet. They are sharing this "map" with the world so others can study it, provided they use it responsibly and don't try to spy on individuals.

In short: They took a chaotic, noisy conversation, organized it into neat trees, taught a robot to read the leaves, and discovered that even in a polarized world, human conversation follows some surprisingly predictable patterns.

1. Problem Statement

Argument mining and stance detection are critical for understanding online discourse, yet existing resources are heavily skewed toward mainstream platforms like Twitter and Reddit. This leaves alt-tech platforms (e.g., Truth Social, Gab, Parler) under-studied, despite their growing influence on political communication.

The Gap: Existing Truth Social datasets consist of isolated posts lacking conversational context (reply trees). Without nested thread structures, it is impossible to study dialogical phenomena like disagreement, persuasion, or stance evolution.
The Challenge: Determining stance in deeply nested conversations is complex. A comment's stance relative to a root post cannot be inferred solely from its immediate text; it requires understanding the chain of parent-child replies. Traditional supervised models often fail to generalize to niche, ideologically homogeneous platforms like Truth Social due to domain shifts.

2. Methodology

The authors developed a comprehensive pipeline involving data collection, enrichment, annotation, and analysis.

A. Data Collection & Augmentation

Source: Built upon an existing public dataset of ~776K Truth Social posts (Feb–Oct 2024).
Preprocessing: Removed empty posts, duplicates, reposts ("retruths"), and posts with fewer than three replies to ensure meaningful interaction.
Augmentation: Used a custom scraper (based on Stanford's framework) and the Truth Social API to retrieve full conversation threads for the filtered posts.
- Retrieved 523,360 comments with in-reply-to IDs to reconstruct directed acyclic trees.
- Enriched data with author-level metadata (follower/following counts, verification status).
Final Corpus: 24,352 root posts and 523,360 comments (2023–2025).

B. Annotation Pipeline

The annotation process was conducted in two stages using a hybrid human-LLM approach:

Argument Mining (AM): A binary classification task to determine if a post contains a claim supported by a premise.
- Human Ground Truth: 750 posts annotated by two coders (Cohen's $\kappa$ = 0.70).
- LLM Scaling: Evaluated LLMs (Gemini-2.5-Flash, GPT-o3, DeepSeek-v3) with various prompting strategies (Few-Shot, Chain-of-Thought, CoT+Few-Shot).
Stance Detection (SD): A 3-class classification task (FOR, AGAINST, NEUTRAL) relative to the immediate parent (post or comment).
- Human Ground Truth: 750 comments annotated (Cohen's $\kappa$ = 0.76).
- Propagation Logic: Stance relative to the root post is inferred by traversing the tree. Note: If an intermediate comment is "NEUTRAL," stance propagation to its descendants is truncated as undefined.
- LLM Scaling: The best-performing model was used to annotate ~107,873 comments.

C. Feature Engineering & Analysis

Content Features: VADER sentiment analysis, Detoxify toxicity scores, and normalized text length.
Engagement Metrics: Likes, replies, retruths, and maximum reply depth.
Topic Modeling: Clustered 28,582 unique hashtags using Qwen embeddings and HDBSCAN to identify emergent topics (e.g., MAGA, Trump, Biden).

3. Key Contributions

TRUTHSTANCE Dataset: The first large-scale, publicly available dataset of Truth Social conversation threads (24K threads, 523K comments) with full reply structure and author metadata.
Human-LLM Benchmark: A gold-standard set of 1,500 human-annotated instances (AM and SD) with inter-annotator agreement, used to evaluate and select optimal LLM prompting strategies.
LLM-Generated Labels: Release of 24,352 argument presence labels and 107,873 stance-to-parent labels generated by the best-performing LLM configuration.
Empirical Insights: Initial quantitative and qualitative analyses of argumentation and stance dynamics on an alt-tech platform.

4. Key Results

A. Model Performance

Argument Mining: All LLMs performed comparably (Macro-F1 $\approx$ 0.82), outperforming the SVM baseline (0.76).
Stance Detection: Gemini-2.5-Flash with CoT+Few-Shot prompting achieved the best performance (Macro-F1: 0.783, Accuracy: 0.836), significantly outperforming GPT-o3 and DeepSeek-v3.
Conclusion: LLMs are robust for this domain, and few-shot prompting with reasoning (CoT) is superior to fine-tuning for out-of-domain generalization.

B. Predictors of Argumentation

Logistic regression analysis revealed that content-based features are the strongest predictors of argumentative posts:

Toxicity: Highly toxic posts are significantly more likely to be labeled as arguments (Odds Ratio: 5.02).
Sentiment: Highly positive sentiment correlates with non-argumentative content (e.g., prayers), while negative sentiment correlates with arguments.
Length: Longer posts are more likely to be arguments.
Engagement: Effects were statistically significant but practically small (e.g., retruths slightly increase argument likelihood).

C. Stance Dynamics

Topic Distribution: The "MAGA" topic generated the highest volume of comments. Stance distribution (Support/Oppose/Neutral) remained relatively consistent across topics, suggesting polarization is a platform-wide characteristic rather than topic-specific.
Depth Evolution: As conversation depth increases (from depth 1 to 6+), the proportion of NEUTRAL responses increases significantly (from 16% to 78.8%), while explicit FOR/AGAINST stances decline. This suggests a shift from debate to conversational maintenance or topic drift in deeper threads.
User Behavior: Users starting with an AGAINST stance tend to maintain it, while those starting with a FOR stance often drift toward NEUTRAL over successive turns.

5. Significance and Applications

Scientific Impact: Fills a critical gap in NLP research by providing data from a politically significant but under-researched alt-tech ecosystem.
Methodological Advance: Demonstrates that LLMs can effectively traverse complex conversational trees to infer stance propagation, offering a scalable alternative to expensive human annotation.
Applications:
- Benchmarking models for platform-specific discourse.
- Studying polarization, disagreement trajectories, and echo chambers.
- Analyzing the correlation between toxicity and argumentation.
- Informing content moderation strategies.
Ethical Considerations: The authors emphasize responsible use, noting risks of political profiling and harassment. The dataset is released under a CC BY-NC 4.0 license for non-commercial research only, with strict privacy safeguards (no specific usernames reported).

Conclusion

The paper successfully introduces TruthStance, a foundational resource for studying political argumentation on alt-tech platforms. By combining high-quality human annotations with scalable LLM inference, the authors provide new empirical evidence that toxicity and sentiment are primary drivers of argumentative content, and that stance polarization evolves distinctively as conversations deepen, often dissolving into neutrality in extended threads.