Technical Summary: SocietyBench

Problem Statement

Current Large Language Model (LLM) and agent benchmarks predominantly focus on task completion (e.g., fixing bugs, navigating GUIs, executing code). While these metrics have matured, they fail to quantify a complementary social capability: how well a model understands and forecasts the dynamics of real-world social events.

Existing forecasting benchmarks suffer from three critical limitations:

1. Narrow Question Types: They often rely on binary yes/no questions (e.g., market prediction markets), missing the nuance of complex social arcs.
2. Data Contamination: Evaluating real events in their original form allows models to score highly via memorization of pre-training data rather than genuine reasoning. Conversely, "live" forecasting benchmarks cannot be fully evaluated until the future unfolds, preventing immediate benchmarking.
3. Neglect of Public Opinion: Social events are driven by a dual layer of factual events and shifting public sentiment/narratives, which most benchmarks ignore.

Furthermore, LLM-driven social simulations (e.g., virtual towns) test fidelity (can the model simulate a society?) rather than forecasting accuracy (can the model predict the outcome of a real event?).

Methodology: SocietyBench

SocietyBench is an end-to-end benchmark designed to evaluate counterfactual social-world forecasting. It transforms real, completed social events into anonymized, counterfactual timelines that preserve causal and temporal structures while removing surface labels that could trigger memory recall.

1. Data Construction & Anonymization

The benchmark covers five heterogeneous events across domains: public controversy, geopolitics, technology policy, financial markets, and trade policy.

• Pipeline: An automated pipeline aggregates Web news and social media posts from five platforms into a date-indexed timeline, separating factual events from a public-opinion layer.
• Three-Phase Anonymization: To prevent pre-training memorization, every timeline undergoes:
  • Entity Substitution: All named entities are replaced using a longest-match lookup table.
  • Date Shifting: Every date is shifted by a constant offset ($\delta$) unique to the event (up to 180 days), destroying absolute calendar positions while preserving inter-date gaps.
  • Adversarial Audit: A panel of adversarial LLMs attempts to re-identify the original event and time period. If confidence is high, the event is rejected or the anonymization is re-run.
• Counterfactual World: The result is a "counterfactual social world" structurally identical to reality but stripped of surface labels, forcing models to rely on forward reasoning rather than recall.

2. Evaluation Protocol

For each event, the timeline is split at multiple cutoff dates ($d_P$). The model acts as a forecaster seeing only the context ($C_P$) before the cutoff and must predict the future ($G_P$).

• Prediction Points: Each event yields 25 prediction points, totaling 125 points across five events.
• Question Types:
  • Calibration Questions: Binary questions asking if event $E$ occurs within a specific time window. The model outputs a probability $\hat{p}$.
  • Temporal Questions: The model predicts the specific calendar date of a future event.
• Scoring Axes (0–100 scale):
  • Probability Calibration: Based on weighted Mean Absolute Error (wMAE) of predicted probabilities against ground truth. A uniform 50% predictor scores exactly 50.
  • Temporal Accuracy: Based on the number of days between the predicted date and the ground truth, normalized by the event's time bucket (0–30, 31–60, 61–90 days). A midpoint guesser scores roughly 50.

Key Contributions

1. SocietyBench Benchmark: A reproducible, end-to-end benchmark featuring five heterogeneous domains, dual-source timelines (news + social media), and a fully automated construction pipeline.
2. Novel Protocol: A design pairing input-side anonymization (entity/date replacement) with output-side dual-axis scoring (calibration and temporal accuracy).
3. Comprehensive Evaluation: Evaluation of 6 frontier LLMs, 3 agent frameworks, and 2 non-LLM baselines across 125 prediction points in both Chinese and English editions.

Experimental Results

The study evaluated six frontier LLMs (GPT-5.5, Gemini-3.5-Flash, Claude-Opus-4.8, DeepSeek-V4-Pro, Kimi-K2.5, Doubao-Seed-2.0-Pro) and three agent frameworks (LangGraph, AutoGen, MiroFish) built on the Doubao base.

Main Findings

• Performance Ceiling: The strongest model, GPT-5.5, achieved an overall score of 75.0/100, significantly above the trivial anchor of 50 but indicating substantial room for improvement.
• Axis Dissociation: The two scoring axes are orthogonal. A model can be strong in calibration but weak in temporal accuracy (e.g., Gemini-3.5-Flash: 73.9 Cal / 67.3 Time) or vice versa (e.g., Claude-Opus-4.8: 62.5 Cal / 70.0 Time).
• Event Variance: Performance gaps vary drastically by event. On the "Trump Tariff" event, the temporal gap between the best (GPT-5.5) and worst (Kimi-K2.5) models reached 21.4 points. This suggests single-event evaluations are unreliable.
• Agent Failure: Three agent frameworks built on the Doubao base failed to improve upon the base model. In fact, orchestration (e.g., AutoGen) slightly degraded performance, particularly on the calibration axis.
• Baseline Comparison: Two model-free heuristics (frequency and momentum baselines) scored between 53.8 and 54.2, trailing all LLMs but confirming that LLMs possess predictive signal beyond simple heuristics.

Ablation Insights

• Question Difficulty: "Growing window" questions (Type A) were the primary driver of difficulty; removing them inflated scores by +7.2 points.
• Bias: Models exhibited a strong bias toward denying false events (scoring 76) compared to confirming true ones (60).
• Cutoff Gradient: Models generally performed better on later cutoffs (seeing more history), validating the benchmark's ability to measure context-based forecasting.

Significance and Claims

The paper argues that SocietyBench provides the first rigorous, quantitative leaderboard for social-event forecasting that avoids data contamination.

• Counterfactual Necessity: By anonymizing events, the benchmark ensures that high scores reflect genuine reasoning about social dynamics rather than retrieval of pre-training memories.
• Multi-Event Requirement: The large variance in performance across different events (up to 21.4 points on a single axis) serves as the primary argument for evaluating models across multiple heterogeneous events rather than relying on a single case study.
• Limitations of Current Systems: The results indicate that current frontier LLMs and agent frameworks are far from mastering social forecasting, with the best models recovering only about half the "headroom" above a random baseline. The failure of agent orchestration to improve base performance suggests that current scaffolding techniques do not yet enhance complex social reasoning.

The authors release all anonymized timelines, question banks, ground truth, and scoring code to facilitate future research in this domain.