💬 NLP

FutureSim: Replaying World Events to Evaluate Adaptive Agents

The paper introduces FutureSim, a benchmark that replays real-world events chronologically to evaluate AI agents' ability to forecast future events and adapt over long time-horizons, revealing significant performance gaps among frontier models and highlighting the need for improved test-time adaptation, memory, and reasoning capabilities.

Original authors: Shashwat Goel, Nikhil Chandak, Arvindh Arun, Ameya Prabhu, Steffen Staab, Moritz Hardt, Maksym Andriushchenko, Jonas Geiping

Published 2026-05-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shashwat Goel, Nikhil Chandak, Arvindh Arun, Ameya Prabhu, Steffen Staab, Moritz Hardt, Maksym Andriushchenko, Jonas Geiping

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict the weather for the next three months. But here's the catch: you can't just look at a forecast app. Instead, you have to act like a detective who only knows what happened up to today, and you have to guess what will happen tomorrow, the next day, and so on, as new news arrives.

This is exactly what the paper FutureSim is about. It's a new "training ground" (or benchmark) designed to test how well AI agents can adapt to a changing world in real-time.

Here is a breakdown of the paper's ideas using simple analogies:

1. The Problem: The "Static" vs. The "Living" World

Most AI tests are like taking a final exam based on a textbook that hasn't changed in years. The AI studies the book, takes the test, and gets a grade. But the real world isn't a static textbook; it's a living, breathing movie that keeps playing.

The authors argue that to truly test if an AI is smart, we need to see if it can watch that movie as it plays, update its predictions every day, and adjust its beliefs when new plot twists (news) happen.

2. The Solution: FutureSim (The "Time-Travel" Simulator)

The authors built a simulation called FutureSim. Think of it as a "Groundhog Day" loop for AI, but instead of reliving the same day, the AI lives through a specific three-month period (January to March 2026) that is after the AI's training data ended.

The Setup: The AI starts with a "knowledge cutoff" (like a student who stopped studying in late 2025).
The Task: The AI is asked to predict real-world events (e.g., "Who will win the election in Nepal?" or "Will a specific sports team win?").
The Mechanism: Every day, the simulation "unlocks" a new batch of real news articles from that specific date in history. The AI can search these articles, read them, and then update its prediction.
The Rule: The AI is strictly forbidden from peeking at the future. It can only see what was known up to that specific day.

3. The Game: Betting on the Future

In this simulation, the AI isn't just answering "Yes" or "No." It's acting like a professional gambler or a weather forecaster.

It has to say: "I think there is a 60% chance X happens, a 30% chance Y happens, and a 10% chance Z happens."
The Score (Brier Skill Score): The paper uses a special scoring system.
- If you are confident and right, you get a high score.
- If you are confident but wrong, you get a negative score (punished heavily).
- If you are unsure and say "I don't know" (abstain), you get a neutral score.
- Analogy: It's better to say "I'm not sure" than to confidently bet your house on the wrong horse.

4. The Results: Who Won the Game?

The authors tested several top-tier AI models (like GPT-5.5, Claude Opus, and others) in this environment.

The Winner: GPT-5.5 came out on top. It was the only model that consistently improved its predictions as the days went by, eventually reaching an accuracy of about 25% (which is impressive for guessing future events).
The Losers: Many other models actually performed worse than if they had just refused to guess at all. They were too confident in their wrong guesses.
The "Harness" Effect: The paper found that how you give the AI tools matters. Some models did poorly with their default settings but improved significantly when the researchers gave them better "instructions" and tools to manage their memory and search for news. It's like giving a student a better study guide; they suddenly perform much better.

5. What Did They Learn? (The "Ablations")

The researchers broke the game down to see what skills the AI actually needed to win:

Memory is Key: If you take away the AI's ability to write notes and remember what it learned yesterday, it gets much worse. It needs to remember past clues to solve today's puzzle.
Search is Crucial: The AI needs to actively look for new information every day. If you freeze the news and don't let it see new articles, its predictions get stuck and wrong.
Thinking Harder Helps: When the researchers told the AI to "think longer and harder" (using more computing power) before answering, it got more accurate.
The "Anchor" Problem: If an AI makes a bad guess early on, it often gets "stuck" on that wrong idea and refuses to change its mind, even when new evidence proves it wrong.

6. Why Does This Matter?

The paper concludes that current AI models are still not great at "long-term adaptation." They are good at answering questions based on what they already know, but they struggle to continuously learn and update their beliefs as the world changes around them.

FutureSim provides a realistic, replayable way to measure this specific skill. It's not about whether the AI knows a fact; it's about whether the AI can act like a human expert who watches the news, updates their mental map of the world every day, and makes better guesses over time.

In short: The paper built a time-traveling newsroom to test if AI can learn to predict the future by watching the present unfold. The results show that while some AI is getting good at this, most still need to learn how to update their beliefs without getting stuck on their first guess.

Technical Summary: FutureSim – Replaying World Events to Evaluate Adaptive Agents

Problem Statement

Current AI agents demonstrate significant success on static benchmarks but struggle to adapt over long horizons in dynamic, open-ended environments. While recent benchmarks (e.g., ARC-AGI-3, VendingBench) offer proxies for adaptation, they often rely on human-designed or model-simulated dynamics that imperfectly mirror real-world societal evolution. A critical gap exists in evaluating whether agents can reason about how the world evolves, update their priors based on new information, and maintain calibrated beliefs over time without the ability to alter the underlying environment. The authors posit that forecasting real-world events beyond a model's knowledge cutoff provides a grounded, reproducible setting to measure this "long-horizon test-time adaptation."

Methodology: The FutureSim Environment

The authors introduce FutureSim, an interactive environment that replays real-world events in chronological order to evaluate adaptive agents.

Environment Design

Temporal Structure: The simulation runs on a daily cadence corresponding to real-world dates (January 1 to March 28, 2026). Time steps correspond to the arrival of new information.
Task State: Agents are presented with 330 forecasting questions derived from news articles (specifically Al Jazeera). Each question includes background information, resolution criteria, and a resolution date. Unlike static benchmarks, agents must submit a probability distribution over multiple possible outcomes they generate themselves, rather than selecting from a fixed set.
Context State: The environment provides a chronological replay of news. Agents have access to a deduplicated snapshot of the Common Crawl News (CCNews) corpus. Crucially, access is strictly gated: agents can only search and read articles published on or before the current simulation date.
Agent Actions: The environment enforces a minimal set of actions:
1. submit_forecast(question_id, outcomes): Register or update a probability distribution for an active question.
2. next_day(): Advance the simulation, triggering the arrival of new articles and the resolution of questions whose dates have passed.
Sandboxing: To prevent information leakage, agents are sandboxed (using bwrap) with no live web access. They interact with a local, date-gated news corpus via a hybrid semantic + keyword search tool (LanceDB with Qwen3 embeddings).

Evaluation Metrics

Brier Skill Score (BSS): Adapted for free-form, multi-outcome predictions. It measures the quality of the probability distribution relative to the ground truth. A score of 1 indicates a confident correct answer, 0 indicates abstention, and negative scores indicate performance worse than abstaining.
Top-1 Accuracy: Measures the correctness of the outcome assigned the highest probability.

Experimental Setup

The authors evaluated frontier agents (GPT 5.5, DeepSeek V4 Pro, Opus 4.6, Qwen3.6 Plus, GLM 5.1) in their native harnesses and a custom "Our Harness" designed to improve orchestration. The custom harness includes features like context consumption feedback, structured memory tools, per-question memory, and procedural scaffolding to encourage iterative updates.

Key Contributions

FutureSim Benchmark: A reproducible, open-ended environment that tests adaptive reasoning on real-world events by replaying world history after the model's knowledge cutoff. It prevents future information leakage while allowing agents to search evolving evidence.
Flexible Evaluation Framework: The environment enforces only two actions, allowing researchers to plug in arbitrary models, harnesses, and event data. It supports the study of long-horizon adaptation, search, memory, and multi-agent dynamics.
Empirical Analysis of Adaptation: The paper provides a comprehensive analysis of how current frontier models perform in long-horizon forecasting, highlighting significant room for improvement even in their native agent harnesses.

Results

Performance Separation: There is a clear separation in capabilities among frontier models. GPT 5.5 (in the Codex harness) performed best, achieving a top-1 accuracy of 25% and a positive Brier Skill Score.
Baseline Deficiencies: Many open-weight models (e.g., Qwen3.6 Plus, GLM 5.1) achieved negative Brier Skill Scores in their recommended harnesses, performing worse than simply abstaining from prediction.
Harness Impact: Custom harness engineering significantly improved performance. For instance, open-weight models that failed in their native harnesses crossed into positive Brier Skill Scores when using the authors' improved harness, demonstrating that better orchestration can elicit test-time adaptation.
Adaptation Dynamics:
- GPT 5.5 showed consistent improvement over time but exhibited overconfidence in incorrect predictions (27.4% of wrong top predictions had $\ge$ 0.5 probability).
- DeepSeek V4 Pro updated predictions frequently, improving substantially from weak initial states, though it sometimes relied on null predictions.
- Memory and Search: Ablation studies confirmed that access to memory and iterative agentic search are critical. Removing memory caused all tested models to perform worse. Similarly, freezing the news corpus (no daily updates) or limiting search to a single query significantly degraded accuracy and calibration.
Human Comparison: GPT 5.5's predictions were often aligned with, and sometimes led, human aggregates on prediction markets (e.g., Polymarket), though it lagged in markets with high human preference aggregation (e.g., UK elections).

Significance and Claims

The authors claim that FutureSim offers a realistic setting to study emerging research directions, including:

Long-horizon test-time adaptation: Measuring how agents update beliefs over months rather than single tasks.
Reasoning under uncertainty: Evaluating the ability to search for evidence and calibrate probabilities in a dynamic information stream.
Memory and Context Management: Investigating how agents retain and utilize information over thousands of turns and context window compactions.
Multi-agent dynamics: Observing how agents adapt when competing or cooperating with others in a shared information environment.

The paper concludes that while current agents act suboptimally in FutureSim, the benchmark provides a necessary lower-bound on capabilities and a flexible platform to measure progress toward agents that can continually adapt and learn in the real world. The authors emphasize that FutureSim is purely predictive; it does not simulate counterfactual worlds where agent actions change the environment, limiting its direct applicability to performative prediction domains but ensuring grounded evaluation of world models.