Technical Summary: SURGE – An Event-Centric Social Media Sentiment Time Series Benchmark

1. Problem Definition

Public events on social media (e.g., natural disasters, political conflicts, technology releases) generate massive, fragmented discussions whose collective dynamics are critical for opinion forecasting and crisis response. Effective predictive modeling requires organizing these discussions into event-level time series that align with the event's lifecycle. However, existing resources fail to meet three simultaneous demands:

1. Event-Level Organization: Many datasets organize data by user or anchor signals to exogenous drivers (e.g., weather) rather than the discussion itself.
2. Interaction Structure: Existing time series often discard the reply and repost topology between posts, despite the fact that social dynamics are tightly coupled with interaction structures.
3. Multimodal Alignment: While multimodal forecasting datasets exist, their text sources (news, financial reports) lack the user-to-user reply chains characteristic of social media, treating text as an exogenous side channel rather than an interactive signal.

Consequently, the predictive value of interaction structure for collective opinion dynamics remains untested, and transferability across event types is difficult to evaluate.

2. Methodology: The SURGE Benchmark

SURGE is a multi-event social media benchmark designed to pair event-level temporal signals with aligned text and interaction structures.

2.1 Data Collection and Processing

• Scope: The dataset covers 67 public events across five categories: Natural Disasters, Political Events, Social Movements, Technology Releases, and Sports/Entertainment.
• Sources: Data is aggregated from Twitter, Reddit, and Threads, comprising 817,442 posts (from an initial raw collection of ~1.26 million).
• Preprocessing:
  • Filtering: Posts are deduplicated and filtered to remove short text, emoji/URL dominance, and non-English content.
  • Active Period Detection: Events are trimmed to their "active periods" to exclude sparse, ambient chatter, ensuring the time series reflects genuine engagement.
  • Sentiment Annotation: Posts are labeled as Positive, Neutral, or Negative using Qwen3-32B in a zero-shot manner. These are mapped to numerical scores $\{+1, 0, -1\}$.
  • Validation: A stratified human verification study on 3,000 posts confirmed a Cohen's $\kappa$ of 0.74 between annotators and an 86.5% match rate with LLM labels, with systematic bias bounded by $|\mu| \le 0.05$.

2.2 Time Series Construction

The benchmark constructs calendar-aligned time series at three temporal granularities: 1-day, 12-hour, and 6-hour.

• Targets: For each time bin $t$, two target variables are derived:
  1. Discussion Intensity ($c_t$): The count of posts in the bin.
  2. Sentiment Polarity ($\bar{s}_t$): The average sentiment score of posts in the bin.
• Normalization: Targets are normalized via per-event z-scoring to handle scale differences across events.
• Textual Views: Each time bin is paired with two textual views derived from the same selected posts:
  • Flat View: Posts linearized chronologically without role markers.
  • Structured View: Posts organized into reply chains with explicit role markers (parent/child), treating reposts as reply-like interactions.

2.3 Evaluation Protocols

SURGE defines four benchmark protocols:

1. Numerical-Only Forecasting: Predicting future values using only historical time series.
2. Text-Augmented Forecasting: Predicting using historical series plus text (tested in No-Text, Flat, and Structured configurations).
3. Structure-Aware Evaluation: A metric ($MAE_{reply}(k\%)$) that evaluates performance specifically on the most reply-dense (high-interaction) time bins.
4. Cross-Category Generalization: A "leave-one-category-out" protocol to test transferability across the five event types.

3. Key Contributions

1. The SURGE Dataset: The first benchmark to simultaneously provide event-level multi-event coverage, pre-built time series, bin-aligned textual views, and explicit social interaction structure (reply/repost edges).
2. Evaluation Framework: A comprehensive set of protocols including structure-aware metrics and cross-category generalization tests, addressing gaps in existing social media and multimodal forecasting datasets.
3. Reference Implementation: A lightweight Structure-Aware Probe (CMA) that utilizes cross-modal attention to consume typed post tokens and thread embeddings, serving as a reference for future interaction-aware models.

4. Experimental Results

The authors evaluated representative time-series forecasting (TSF) models (linear, Transformer, CNN, LLM-based, and diffusion-based) and multimodal baselines.

• Strong Local Persistence: Naive baselines (Last Value, Moving Average) remain difficult to beat under Mean Absolute Error (MAE) across all granularities. This indicates a "strong local-persistence regime" where the best short-horizon forecast is often the most recent observation. Learned models often reduce Mean Squared Error (MSE) by smoothing predictions but fail to improve MAE.
• Limited Transfer of Text-Augmented Models: Existing multimodal TSF models (designed for news/financial text) show limited robustness when applied to social media data. They do not consistently improve over numerical-only baselines, and their performance is insensitive to the structural form of the text (Flat vs. Structured).
• Interaction Density Matters: The structure-aware metric ($MAE_{reply}$) reveals that reply-dense periods are intrinsically harder to forecast. Model rankings on these dense slices diverge from rankings on aggregate metrics, suggesting that high-interaction periods are where models fail most.
• Generalization Asymmetry: Cross-category transfer is structurally asymmetric. Natural disasters are the hardest category to generalize to, while social movements are the easiest. The difficulty varies significantly by target variable (Intensity vs. Sentiment).

5. Significance and Claims

The paper positions SURGE not as a proposal for a new forecasting model, but as a stress test for event-driven social media forecasting. Its significance lies in exposing three fundamental properties of the domain:

1. Persistence is strong but insufficient: Current optimization objectives (MSE) may suppress large deviations without improving central-tendency accuracy (MAE) on event-driven series.
2. Text is not automatically useful: The internal structure of social media text (reply chains) differs fundamentally from the unidirectional text in financial/news datasets. Multimodal models must be explicitly designed to handle this interaction structure to be effective.
3. Evaluation regimes differ by density: Aggregate metrics obscure model failures in high-interaction periods. The benchmark provides axes (interaction density and category shift) along which existing methods fall short, particularly in scenarios most critical for crisis response and opinion monitoring.

The authors conclude that SURGE provides the necessary data, protocols, and diagnostic measurements to advance research into interaction-aware forecasting, leaving the design of stronger models as a task for future work.