Technical Summary: WORLDREASONER

Problem Statement

Forecasting real-world events requires Large Language Model (LLM) agents to reason under uncertainty using incomplete, time-bounded information. Current evaluation methods face two primary challenges:

1. Temporal Data Leakage: Static benchmarks (e.g., Autocast) become contaminated as model training corpora advance. An agent answering a question about a past event may be recalling memorized training facts rather than genuinely forecasting, rendering outcome accuracy an unreliable measure of predictive reasoning.
2. Incomplete Evaluation Metrics: Even with clean temporal boundaries, most benchmarks score only the final resolved answer. A model may achieve high accuracy by recalling facts, citing hallucinated evidence, or constructing unsupported causal stories. Conversely, a model may retrieve relevant evidence but fail to convert it into a calibrated probability. Forecasting is inherently an evidence-grounding and causal-reasoning task, yet current evaluations often collapse these dimensions into a single scalar.

Methodology

The paper introduces WORLDREASONER, a reproducible evaluation framework for historical event forecasting that enforces strict temporal constraints and evaluates agents across three complementary axes.

1. Framework Design

• Simulated Historical Replay: Each task presents a resolved real-world question at a specific simulated forecast date. The agent operates within a Temporal Gateway that restricts all tool access to evidence (articles, events) available strictly before that date. This separates evaluation time from historical time, ensuring reproducibility.
• Three-Axis Evaluation: Instead of a single score, the framework evaluates submissions on:
  • Outcome Quality: Accuracy, Brier score, and log score against the resolved answer.
  • Evidence Quality: Whether cited or accessed sources are time-valid and relevant, measured by source precision against a hindsight evidence corpus.
  • Reasoning Quality: Whether the agent's forecast graph recovers hindsight-validated causal events, measured by key-event recall against a reference graph.

2. Agentic Benchmark Construction Pipeline

The benchmark is constructed via a scalable, two-pipeline system:

• Forward Pipeline: Ingests news streams and prediction markets (e.g., Polymarket). An LLM-driven generator proposes forecasting questions, which are filtered for clarity, measurability, and temporal validity.
• Backward Pipeline: After an event resolves, a Hindsight Agent collects post-resolution evidence to synthesize a causal explanation. A GraphBuilderAgent converts this into a structured Directed Acyclic Graph (DAG) of dated events and causal edges. These graphs serve as the "ground truth" references for reasoning evaluation.
• Scale: The resulting benchmark contains 345 resolved tasks derived from 14,141 articles, with hindsight graphs covering 8,087 extracted events across 10 domains.

3. Experimental Setup

The authors evaluated representative frontier models (Gemini 3, DeepSeek V4, GPT-4o, GPT-5.4) across six controlled agent settings:

1. Vanilla LLM: Parametric knowledge only.
2. Causal Simulation: Internal causal graph reasoning without retrieval.
3. Search-Enabled: Simulated-date retrieval.
4. Search-Enabled Graph: Retrieval combined with explicit graph construction.
5. Near-Resolution: Access to evidence one day before resolution.
6. Real-Time Agent: Live web access (non-reproducible upper bound).

A contamination filter was applied to exclude model-question pairs where the question's start date preceded the model's knowledge cutoff, preventing the conflation of memorization with forecasting.

Key Results

• Retrieval is the Primary Driver: Temporally valid retrieval is the strongest driver of outcome accuracy. While Causal Simulation alone did not improve over Vanilla LLM (56.6% vs. 58.7% weighted accuracy), Search-Enabled agents reached 68.8%.
• Graph Construction vs. Answer Selection: Explicit causal graph construction improves key-event recovery (Reasoning Quality) but does not automatically guarantee better answer selection (Outcome Quality). Search-Enabled Graph agents achieved 9.8% Key-Event Recall (KER) but trailed Search-Only agents in accuracy (64.4% vs. 68.8%).
• Grounding and Correctness: Correct forecasts in the Search-Enabled Graph setting showed significantly higher source precision (62.6% vs. 48.0%) and key-event recall (10.8% vs. 8.0%) compared to incorrect forecasts. However, a gap remains: agents can retrieve relevant evidence and identify causal events yet still fail to convert them into calibrated probabilities.
• Temporal Sensitivity: Accuracy increases as the simulated forecast date approaches the resolution date, confirming that the benchmark rewards later access to evidence and that early forecasts are genuinely harder.

Significance and Claims

The paper claims that WORLDREASONER addresses the trade-off between contamination control and experimental practicality by enabling reproducible, simulated-date replay with post-resolution reasoning references.

The authors emphasize that three-axis evaluation catches what accuracy misses. A single leaderboard score cannot distinguish between an agent that is correct due to time-valid evidence retrieval, causal chain reconstruction, or mere memorization of post-resolution facts. Conversely, it cannot reveal cases where an agent retrieves useful evidence but fails at the final probability conversion step.

The paper concludes that while temporally situated evidence access substantially improves outcome quality, and graph construction improves reasoning alignment, the ability to convert grounded causal evidence into calibrated probabilities remains an open challenge. The framework is presented as a research evaluation tool to diagnose these specific failure modes rather than a decision-support system for high-stakes forecasting.