Technical Summary: Agentic Time Machine as an Infrastructure for Future-Event Forecasting

1. Problem Statement

Forecasting real-world future events (e.g., election outcomes, monetary policy decisions, financial market movements) presents a unique challenge for Large Language Model (LLM) agents. Unlike standard information retrieval tasks, the "correct" answer does not exist in any online source at the time of prediction. Agents must synthesize noisy evidence, account for uncertainty, and reason about potential outcomes without access to the resolved ground truth.

Current evaluation paradigms face a fundamental trade-off between efficiency and environment fidelity:

• Live Benchmarks: Systems like FutureX, Prophet Arena, and ForecastBench offer dynamic web access and open-ended tool use. However, they suffer from inherently slow feedback loops, requiring researchers to wait days or weeks for real-world events to resolve before scoring predictions.
• Retrospective/Frozen Benchmarks: To accelerate development, some approaches replay historical questions using static, pre-frozen databases or closed-book inference. While these provide fast feedback, they sacrifice environmental realism by restricting agents to limited, non-dynamic data sources, failing to simulate the open-web interactions required in actual deployments. Furthermore, naive retrospective evaluation is vulnerable to answer leakage, where updated web snippets or pages inadvertently reveal the resolved outcome.

2. Methodology

The paper proposes two core components to address these challenges: a new evaluation infrastructure and a multi-agent forecasting framework.

2.1. Agentic Time Machine (TM)

The Agentic Time Machine is an LLM-based evaluation infrastructure designed to enable fast, reproducible local replay of forecasting tasks while preventing answer leakage.

• Mechanism: TM sits between the forecasting agent and live web tools (search and page fetching). It employs an LLM-based leakage filter ($\phi$) that processes tool outputs before they reach the agent.
• Filtering Logic: The filter evaluates each retrieved item (snippet or page content) against two criteria:
  1. Direct Answer Leak: Does the content explicitly state or strongly imply the ground-truth answer ($a^\star$)?
  2. Post-Date Info Leak: Does the content contain time signals (publish dates, update timestamps, event dates) later than the question's cutoff time ($t$)?
• Rationale for LLM Filtering: Simple rule-based date filtering is insufficient because web content often evolves on stable URLs (e.g., historical stock price pages where the URL date is old, but the data table updates daily). The LLM filter inspects the actual content to distinguish between pre-cutoff background information and post-cutoff revelations.
• Outcome: TM approximates the web state as it existed at time $t$, allowing agents to interact with a dynamic, open-ended web environment without access to the resolved answer.

2.2. Planner-Solver-Aggregator Multi-Agent Framework

To improve forecasting performance, the authors propose a multi-agent system that decomposes the reasoning process without requiring additional training or tools beyond a single-agent baseline.

• Planner: Analyzes the prediction question, performs exploratory searches, and decomposes the problem into 2–4 distinct analytical angles (e.g., historical base rates, causal drivers, market risk, campaign strength). It generates a JSON plan defining the role and search focus for each solver.
• Solvers: Multiple solver agents run in parallel, each assigned a specific angle by the planner. They actively use web tools to gather evidence along their designated paths and produce structured JSON reports grounded in evidence rather than priors.
• Aggregator: Receives the planner's plan and all solver reports. It identifies points of agreement and conflict, performs additional targeted searches if necessary, and synthesizes a final grounded forecast.
• Design Philosophy: This approach diversifies the evidence-gathering process itself, rather than just diversifying the final answers (as in self-consistency). It allows the system to cover more relevant evidence slices without forcing a single agent to pursue every possible reasoning line.

3. Key Contributions

1. Agentic Time Machine Infrastructure: A novel evaluation system that simulates the open web at a specific historical point in time. It enables fast, local replay of forecasting tasks while explicitly blocking post-cutoff content and answer-revealing information, effectively bridging the gap between live realism and offline efficiency.
2. Planner-Solver-Aggregator Framework: A zero-shot, multi-agent architecture that decomposes forecasting tasks into diverse analytical perspectives. It utilizes parallel evidence gathering and a reconciliation step to produce a single forecast, outperforming standard self-consistency baselines.
3. Empirical Validation: Comprehensive evidence demonstrating that:
   • TM effectively reduces information leakage, significantly lowering scores on tasks where agents previously relied on direct answer retrieval.
   • Offline scores achieved under TM correlate strongly ($r=0.92$) with live leaderboard performance, validating TM as a reliable proxy for real-world testing.
   • The proposed multi-agent framework achieves state-of-the-art performance on both offline datasets and the live FutureX leaderboard.

4. Experimental Results

4.1. Validation of Agentic Time Machine

• Leakage Reduction: When applied to five different LLM backbones (including GPT-5.4, GLM-5, and Kimi-K2.5) on a 61-question subset (FutureX-Past), enabling TM reduced scores by 25 to 37 points. This drop confirms that previous high scores were inflated by answer leakage from the live web.
• Difficulty Sensitivity: The score drop was most pronounced on Level 4 (quantitative/open-ended) questions, where direct retrieval is impossible without leakage. Level 1 (few-option) questions showed a smaller drop, as models could still rely on historical baselines and elimination strategies.
• Correlation with Live Performance: Offline scores under TM showed a strong Pearson correlation ($r=0.92$) with live FutureX scores, confirming that TM provides a faithful simulation of real-world forecasting constraints.

4.2. Performance of the Multi-Agent Framework

• Offline Benchmarks (FutureX-Past & Polymarket):
  • On FutureX-Past, the framework achieved a weighted score of 39.51, outperforming the best self-consistency baseline (SC@5, 36.81) and the strongest single-agent tool-augmented baseline (35.02).
  • On Polymarket (80 multi-option questions), the framework scored 64.99, surpassing SC@5 (62.26) and the top single-agent baseline (59.24).
• Live Leaderboard (FutureX):
  • The system (listed as FutureGPTv1) participated in four consecutive weekly rounds (April W3 to May W2, 2026).
  • It achieved ranks of #4, #2, #1, and #6, resulting in the best average rank (3.25) among the compared public systems (MiroFlow and H2O AI Super Agent).
  • As of June 17, 2026, it ranked 1st on FutureX's official eight-week overall leaderboard with a Bradley-Terry rating of 4.61.
• Efficiency: The framework was more efficient than self-consistency baselines. While SC@5 required 81.0 tool-call rounds on average (on Polymarket) to gain only 1.27 points over SC@3, the proposed framework achieved higher accuracy with 47.1 rounds, demonstrating that role decomposition is more efficient than repeated sampling.

5. Significance and Claims

The paper claims that Agentic Time Machine provides a critical infrastructure for the field of LLM forecasting. By enabling fast, leakage-free local replay, it allows researchers to iterate on agent designs without the prohibitive time costs of waiting for real-world event resolution. The strong correlation between offline TM scores and live leaderboard performance validates TM as a reliable proxy for evaluating forecasting capabilities under genuine uncertainty.

Furthermore, the planner-solver-aggregator framework demonstrates that decomposing complex forecasting problems into diverse analytical angles and reconciling parallel evidence streams yields superior performance compared to both single-agent baselines and simple self-consistency voting. The authors emphasize that these improvements are achieved without fine-tuning or additional tools, suggesting that architectural decomposition is a key lever for advancing agent-based forecasting.

The work concludes that the combination of a contamination-free evaluation environment (TM) and a structured multi-agent reasoning framework represents a significant step toward robust, real-world future-event forecasting agents.