💻 computer science

ForecastBench-Sim: A Simulated-World Forecasting Benchmark

The paper introduces ForecastBench-Sim, a novel forecasting benchmark built on Freeciv game simulations that overcomes real-world constraints by enabling the generation of immediately resolvable, controlled, and diverse forecasting tasks to study probabilistic reasoning in dynamic environments.

Original authors: Jaeho Lee, Nick Merrill, Ezra Karger

Published 2026-06-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jaeho Lee, Nick Merrill, Ezra Karger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to test how good a weather forecaster is. In the real world, you'd have to wait days or weeks to see if they were right. If you want to test if they can predict a "once-in-a-century storm," you might have to wait a century. If you want to ask, "What if the storm hit the city instead of the coast?" you can't answer that, because the storm already happened one way, and you can't rewind time to see the other path.

ForecastBench-Sim is a new tool created by researchers to solve these problems. Instead of waiting for real life to play out, they built a digital sandbox using a strategy game called Freeciv (similar to the famous Civilization series). Think of it as a "flight simulator" for predicting the future.

Here is how it works, broken down into simple concepts:

1. The "Frozen Snapshot" Game

Imagine you are looking at a paused video game screen. You see a map with cities, armies, and gold treasuries. This is the "World Report."

The Task: You (or an AI) have to make predictions about what will happen next. Will a specific city grow? Will a country run out of money? Will two nations go to war?
The Twist: You don't get to see the future. The game is paused, and the "future" is hidden in a locked box.
The Reveal: Once you make your guess, the researchers hit "Play." The game runs forward automatically. They open the box, check the actual outcome, and grade your prediction immediately.

2. Why This is a "Superpower" for Testing

In the real world, testing predictions is slow and messy. In this game world, the researchers have "God mode" controls that let them do three special things:

The "Rewind" Button (Interventions):
In real life, you can't test "What if?" questions easily. In this game, they can take a saved game, change one tiny thing (like giving a country 500 extra gold coins or changing its government type), and then run the game forward twice.
- Scenario A: The country keeps its old government.
- Scenario B: The country gets the new government.
  They can then ask the AI: "How did that change affect the outcome?" This lets them test if the AI understands cause and effect, not just patterns.
The "Fast-Forward" Button (Rare Events):
Real-world disasters (like a massive economic crash) are rare. You can't wait for them to happen often enough to test an AI. In the game, the researchers can force a "disaster" to happen 100 times in a row to see how well the AI predicts these rare, scary events.
The "Instant Grade" Button:
There is no waiting. The game runs, the result appears, and the score is calculated instantly. This allows them to test thousands of predictions in the time it would take to wait for one real-world event to resolve.

3. What They Tested

The researchers used this tool to test various AI models (like GPT-5, o3, and others).

The Results: They found that the AI models got better at predicting things that were happening soon (like 30 turns ahead) and worse at predicting things far in the future (like 210 turns ahead). This is exactly what you'd expect from a human: it's easier to guess what happens next week than what happens next year.
The Check: They also made sure the AI wasn't just "cheating" by misreading the game report. They gave the AI simple questions about the current state (like "How many cities does Country X have right now?"), and the AI got almost 100% of those right. This proved that when the AI made mistakes about the future, it was because the future is hard to predict, not because it couldn't read the instructions.

4. The Bottom Line

The authors are very clear: This game is not a replacement for real-world testing. It's a training gym.

Just as a pilot trains in a flight simulator to learn how to handle storms without crashing a real plane, AI systems can use ForecastBench-Sim to practice their "probabilistic reasoning" (guessing the future with numbers) in a safe, controlled environment. It helps researchers understand how AI handles uncertainty, cause-and-effect, and rare disasters, providing a quick and clean way to see how these systems are improving before they are used in the messy, slow-moving real world.

Technical Summary: ForecastBench-Sim

Problem Statement

General-purpose AI forecasting benchmarks, such as the existing ForecastBench, face structural limitations inherent to the real world. These include slow resolution times for outcomes, the scarcity of rare "tail" events, and the inability to score counterfactual questions since only one historical trajectory is realized. These constraints introduce methodological pitfalls, such as the difficulty of evaluating probabilistic reasoning under uncertainty when ground truth is delayed or singular. While "pastcasting" (reusing unseen past events) offers a partial solution, it lacks the control over interventions and resolution timing that a simulated environment provides.

Methodology

The authors introduce ForecastBench-Sim, a benchmark built upon Freeciv (a turn-based strategy game modeled on the Civilization series) and the CivRealm infrastructure. The system operates as follows:

World Generation & Reporting: The benchmark utilizes Freeciv game rollouts. A fixed "world report" is generated at Turn 60, summarizing the state of civilizations (cities, technology, territory, treasury, diplomacy, and conflicts) in structured text, tables, and maps. This report serves as the input for forecasters (both LLMs and humans), hiding all future turns.
Forecasting Tasks: Based on the Turn 60 report, the benchmark generates two types of questions across multiple horizons (H1–H7, corresponding to 30-turn increments up to Turn 270):
- Binary Questions: Asking whether specific events or relations will hold (e.g., "Will Treasury > X?").
- Continuous Questions: Asking for specific future values of world variables (e.g., exact treasury count), eliciting five quantiles (p10, p25, p50, p75, p90).
- Comprehension Checks (H0): Questions about the Turn 60 snapshot itself to verify report reading capabilities.
Scoring Protocols:
- Binary Forecasts: Evaluated using the Brier score.
- Continuous Forecasts: Evaluated using the Continuous Ranked Probability Score (CRPS) computed from elicited quantiles. Scores are normalized by fixed per-family value ranges (e.g., 2000 for treasury) to allow averaging across different variable types.
Evaluation Regimes: The benchmark supports three distinct conditioning regimes:
1. Unconditional: Standard forecasting from a fixed report ( $P(Y)$ ).
2. Observational Conditional: Targeting $P(Y|X)$ by sampling natural variations across a corpus of simulated worlds.
3. Interventional: Targeting $P(Y|do(X))$ by mutating the savegame state (e.g., changing government type or adding gold) before rolling out the simulation. This allows for "fork-and-resolve" causal questions where separate ground truths are generated for each branch.

Key Contributions

Simulated-World Substrate: The creation of a benchmark that preserves core forecasting structures (partial information, path dependence, interacting agents) while granting evaluators full control over resolution and interventions.
Interventional Scoring: The ability to score counterfactual and causal questions by generating paired intervention worlds, a capability impossible in real-world benchmarks where only one history exists.
Tail-Risk Evaluation: The capacity to densely sample disruptive or rare outcomes (e.g., economic crashes) rather than waiting for them to occur naturally, facilitating the study of tail-risk forecasting.
Comprehensive Artifact Release: The release includes the benchmark pipeline, question families, scoring protocols, and validation slices from model evaluations and an anonymized human pilot.

Results

The paper reports validation results from a curated subset of existing model runs (covering 9 models) and a small human pilot:

Model Performance:
- Binary (H1–H7): Mean Brier scores ranged from 0.220 (GPT-5.1) to 0.313 (Gemini 2.5 Flash).
- Continuous (H1–H7): Mean normalized CRPS ranged from 0.283 (o3) to 0.590 (Gemini 2.5 Pro).
- Correlation: ForecastBench-Sim rankings showed moderate positive correlation with the real-world ForecastBench Dataset (Spearman $\rho = +0.43$ ) and the Epoch Capabilities Index ( $|\rho| = 0.48$ ), suggesting the benchmark captures general forecasting ability.
Horizon Dependence: Performance degraded as the forecast horizon extended. Mean Brier scores increased from 0.205 (H1) to 0.264 (H7), and CRPS increased monotonically by a factor of 4.8, indicating the tasks contain forecastable signal that diminishes with time.
Comprehension: All curated models achieved near-perfect scores on H0 checks (Brier < 0.032), confirming that errors in forward-looking tasks were not due to an inability to read the Turn 60 reports.
Interventional Gains: In pilot interventions (e.g., switching to a Republic government or adding 500 gold), models showed positive mean intervention gains. A placebo control (null-conditional wording) showed negligible changes, suggesting models were responding to the actual intervention rather than just conditional phrasing.
Human Pilot: Ten human participants answered continuous questions, yielding results close to a uniform-bin baseline, serving as a feasibility check for future larger-scale human baselines.

Significance and Claims

The authors position ForecastBench-Sim not as a final empirical ranking of forecasters, but as a complementary tool to real-world benchmarks. Its primary significance lies in providing a controlled, immediately resolvable environment for studying:

Probabilistic reasoning under dynamic world states.
Calibration and consistency in language model forecasting.
Causal updating and counterfactual reasoning (via the interventional regime).
Tail-risk behavior through dense sampling of disruptive outcomes.

The paper acknowledges limitations, noting that Freeciv worlds are simplified and stylized, and that the current human study is merely a pilot. The authors emphasize that the benchmark is intended to be a reusable evaluation substrate for narrower claims about specific forecasting behaviors, rather than a definitive leaderboard for general AI capabilities.