Stance Detection in Prediction Markets: Addressing Imbalanced Trader Commentary via Counterfactual Augmentation and Market Context

Technical Summary: Stance Detection in Prediction Markets

Problem Statement
Prediction markets, such as Polymarket, aggregate crowd beliefs into real-time probability estimates. While market prices reflect aggregate sentiment, the trader comments posted beneath these markets contain rich, directional stance signals that prices alone cannot capture. Extracting these signals presents three compounding challenges:

1. Extreme Brevity: Comments are often single phrases or sentences.
2. Domain-Specific Vernacular: The text relies heavily on trader-specific slang (e.g., "this is cooked," "free money") distinct from the political or journalistic corpora typically used to train stance detection models.
3. Severe Class Imbalance: The dataset exhibits a significant skew where only 8.7% of comments express opposition (Anti Position) to the market outcome, rendering standard accuracy-based evaluation misleading and necessitating targeted minority-class strategies.

Methodology
The study introduces a comprehensive framework involving data construction, input transformation, and synthetic augmentation, evaluated through a $4 \times 3$ ablation study using a fine-tuned RoBERTa-base classifier.

• Dataset Construction: The authors constructed and released the first annotated dataset for this domain, comprising 2,229 human-annotated comments across 12 Polymarket markets (politics, sports, finance). The distribution is 63.1% Neutral, 28.2% Pro Position, and 8.7% Anti Position.
• Market Context Input Transformation: To address ambiguity in short comments, the market question is prepended to each comment (e.g., "Market: {question} Comment: {text}"). This provides zero-cost grounding. Additionally, Named Entity Recognition (NER) is used to mask specific entities (PERSON, ORG, etc.) with a generic ENTITY token to prevent the model from learning entity-specific shortcuts.
• LLM-Driven Counterfactual Augmentation: To mitigate the scarcity of Anti samples (194 total), the authors employed the Anthropic API (Claude Haiku 4.5) to generate synthetic Anti comments by flipping existing Pro comments. The generation process preserves trader tone, slang, and length. Two augmentation doses were tested: 50% and 100% of the generated samples added to the training set.
• Experimental Design: The study evaluated four model configurations ({2-class, 3-class} $\times$ {with/without context}) across three augmentation conditions (baseline, 50% synthetic, 100% synthetic). The primary evaluation metric was Macro F1, with specific focus on Anti recall and Anti F1.

Key Results
The ablation study yielded three primary findings:

1. Market Context is the Dominant Factor: Providing the market question as input context was the single most impactful intervention. In the 3-class setup, adding context increased Anti recall from 0.10 to 0.45 and Macro F1 from 0.42 to 0.54. The best overall performance (Macro F1 = 0.68) was achieved by the 2-class configuration (Pro/Anti only) with context.
2. Conditional Effectiveness of Augmentation: Counterfactual augmentation showed a non-linear, dose-dependent relationship with performance.
   • Benefit in Weak Configurations: In the 3-class baseline (where Anti recall was 0.10), 50% augmentation improved Anti F1 from 0.10 to 0.24.
   • Degradation in Strong Configurations: In the strong 2-class-with-context configuration, 100% augmentation degraded performance significantly, dropping Macro F1 from 0.68 to 0.50 (a 26% relative decrease).
3. Optimal Dose is 50%: Across configurations, 50% augmentation consistently outperformed 100% augmentation. Full-dose augmentation introduced noise that corrupted the decision boundary, particularly in configurations where the baseline was already strong.

Analysis and Interpretability
Attention map analysis provided mechanistic explanations for these results:

• Context: Without market context, models attended primarily to start/end tokens (length proxies). With context, attention distributed across substantive tokens in both the market question and the comment, resolving ambiguity.
• Augmentation Failure Modes: Synthetic Anti comments tended to be longer and more grammatically complete than real trader opposition (which is often terse and slang-heavy). In strong configurations, this distribution mismatch caused the model to shift attention away from stance-bearing tokens toward synthetic-pattern artifacts, leading to misclassification of real Pro comments as Anti.

Significance and Claims
This work represents the first stance detection study applied specifically to prediction market commentary. Its significance lies in:

• Resource Creation: Providing the first annotated dataset and trained checkpoints for this specific domain.
• Methodological Insight: Demonstrating that for short, slang-heavy financial text, input grounding (market context) is more critical than data augmentation.
• Augmentation Strategy: Establishing that LLM-driven counterfactual augmentation is not universally beneficial; its efficacy is conditional on the baseline model's strength and the fidelity of the synthetic data to the target domain's stylistic distribution. The paper concludes that augmentation should be applied cautiously at conservative doses (50%) and validated against held-out real data before deployment.