💬 NLP

Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters

This paper introduces "Hindcast," a rigorous evaluation framework that prevents data leakage in LLM forecasting by replaying prediction markets against a frozen, time-cutoff snapshot of public information to ensure models are graded on genuine foresight rather than post-event recall.

Original authors: Xiao Ye, Jacob Dineen, Evan Zhu, Shijie Lu, Kevin Song, Ben Zhou

Published 2026-07-16

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiao Ye, Jacob Dineen, Evan Zhu, Shijie Lu, Kevin Song, Ben Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to test how good a detective is at solving a mystery. You give them a case file and ask, "Who stole the cookie?" If the detective simply opens the file and reads the answer written at the very bottom, they aren't really showing off their detective skills; they are just showing off their ability to read. This is the tricky problem scientists face when testing Artificial Intelligence (AI) models on predicting the future.

To see if an AI is truly smart, researchers usually ask it to predict things that have already happened, like "Who won the 2022 World Cup?" The problem is that today's AI models have been trained on the entire internet, including articles written after the game ended. So, when the AI says, "Argentina won," it might not be because it figured out the clues beforehand; it might just be remembering the final score from its training data. It's like a student taking a history test who has secretly memorized the answer key. To fix this, scientists need a way to test the AI's "foresight" (guessing what will happen) without letting it cheat by looking at the "answer key" (what actually happened).

This is where a new study called HINDCAST comes in. The researchers wanted to see if AI could actually predict the future if we forced it to stand in the past, with no knowledge of what happened next. They set up a time-travel simulation using prediction markets (places where people bet on outcomes) and a frozen archive of the internet. They found that when you stop the AI from cheating, it can still get better at guessing by reading old news, but only if that news actually had useful facts. If the old news was just people guessing and shouting opinions, reading it actually made the AI worse at predicting the truth.

The Time-Travel Test

The researchers built a system they call HINDCAST (a play on "forecast," but looking backward). Imagine a time machine that freezes a specific moment in history, say, one month before a big sports tournament starts. At this frozen moment, the AI is allowed to look at a library of news articles and forum posts, but only the ones written before that date. It cannot see anything written after that moment, even if it's sitting in the library today.

To test the AI, the researchers used real-world betting markets called Polymarket. These are like digital casinos where people bet on whether something will happen (like "Will Team X win?"). Because these markets have already finished, the researchers know the true answer. They also know what the "market price" was at that frozen moment in the past, which represents what a crowd of humans thought the odds were back then.

The setup works like this:

Freeze Time: Pick a past date ( $t_0$ ) before a market resolved.
Lock the Library: The AI can only search a frozen snapshot of Reddit (a popular internet forum) from before that date.
The Prediction: The AI reads the available posts and guesses the outcome.
The Score: The AI gets graded on two things: how close it was to the actual final result, and how close it was to what the human bettors thought at that exact moment.

The Big Discovery: Facts vs. Hype

The team tested nine different AI models to see if giving them access to this "frozen library" helped them predict better than just guessing from their own memory.

The Good News: For most of the AI models, reading the old posts did help. In fact, for eight out of the nine models, the AI made fewer mistakes when it could read the archive. The biggest improvement was seen in a model called Qwen3-32B, which reduced its error rate by 23%. This suggests that when there are real facts available in the past, AI can use them to make smarter guesses.

The Bad News (and the Catch): The help wasn't universal. It depended entirely on what the AI was reading.

Where it worked: In topics like Sports, Awards, and Trading, the internet posts before the event were full of concrete facts (e.g., "The team's star player is injured" or "The price of gold is rising"). In these cases, the AI read the facts and got better at predicting the winner.
Where it failed: In topics like Entertainment (like guessing which song will be #1) or Politics, the internet was full of loud opinions, fan hype, and speculation. When the AI read these posts, it got confused. It started believing the "hype" was a fact. For example, if fans were screaming that a new song would be #1, the AI would bet on it, even though most songs never actually make it to the top. In these cases, reading the archive made the AI worse than if it had just ignored the internet and stuck to its own logic.

The "Over-Reading" Problem

The study found a funny pattern: the more time a market stayed open, the more likely the AI was to get confused by the noise. If a betting market was open for a long time, the internet filled up with endless chatter and conflicting opinions. The AI, trying to be helpful, read all of it and started to overthink, eventually making bad guesses based on loud opinions rather than solid facts.

One specific model, R1-Distill-Qwen-7B, actually got worse overall when it was allowed to read. The researchers think this happened because the model got so bogged down in reading long, confusing chains of reasoning that it forgot the actual evidence. It's like a student who reads so many different opinions on a topic that they forget the simple facts and end up guessing wrong.

What This Means for the Future

The main takeaway from HINDCAST is that an AI's ability to predict the future is only as good as the quality of the information it can find before the event happens. If the internet is full of facts, the AI can be a great forecaster. If the internet is full of noise and hype, the AI might just become a confident fool, believing the loudest voice instead of the truth.

The researchers also showed that this "time-travel" testing method is a powerful tool. Because the library is frozen, you can test new AI models against the same old questions without them cheating. This means we can keep improving our tests as AI gets smarter, ensuring that we are really measuring how well they think, not just how well they remember.

In short, HINDCAST proves that while giving AI a library of facts helps it see the future, giving it a library of rumors might just make it blind.

Technical Summary: HINDCAST – Replaying Prediction Markets to Evaluate LLM Forecasters

1. Problem Statement

Current evaluation methods for Large Language Model (LLM) forecasters rely on backtesting resolved questions, where models are graded on the probability they assign to known outcomes. The authors identify two critical "leaks" in this paradigm that allow models to cheat by accessing information that would not have been available at the time of the forecast:

Retrieval Leakage: Models equipped with retrieval systems can surface reports, articles, or discussions written after the event occurred, effectively turning forecasting into a lookup task.
Training Data Leakage: As models are retrained on data closer to the present, questions that were in the future for previous model versions become part of the training data for newer versions.

Consequently, standard evaluations often measure a model's recall of the outcome rather than its foresight (the counterfactual probability it would have assigned at a past time $t_0$ ). While live benchmarks avoid this by posing open questions, they lack the immediate ground truth and the human baseline (market price) provided by resolved questions.

2. Methodology: HINDCAST

The authors introduce HINDCAST, an evaluation protocol designed to close these leaks by simulating a forecast made at a specific past date $t_0$ , before the outcome was known and before any post-event information existed in the retrieval corpus.

Core Components

Frozen Information Environment: The system uses a temporally pinned archive of public Reddit (derived from Pushshift monthly snapshots). This archive is immutable; once a snapshot is captured, it never changes. This ensures that retrieval at any simulated time $t_0$ only returns content created strictly before $t_0$ .
Prediction Market Ground Truth: The evaluation uses resolved Polymarket binary markets. For each market, the system records:
- The resolved outcome ( $y$ ).
- The market-implied probability at time $t_0$ ( $q_{t_0}$ ), which serves as a human forecast baseline derived from the same pre- $t_0$ information.
Cutoff Enforcement: To prevent routing errors, the retrieval backend enforces a strict created_at < t_0 predicate at the SQL level for all search and lookup operations.

Evaluation Pipelines

The study compares two predictor families against the same interface:

Zero-shot Baseline: The LLM forecasts using only its parametric memory, instructed to rely only on knowledge available at $t_0$ .
Retrieval Agent: The LLM operates in a bounded research loop, querying the frozen archive at $t_0$ , reading evidence, and refining its forecast.

Metrics

Forecasts are scored against two targets:

Outcome Recovery: Accuracy and Brier score against the resolved binary outcome ( $y$ ).
Market Improvement: The absolute difference between the model's forecast and the market's price at $t_0$ ( $|\hat{p}_{t_0} - q_{t_0}|$ ).

3. Key Contributions

HINDCAST Protocol: An evaluation framework that measures a model's forecast at a past $t_0$ on resolved questions while holding retrievable information fixed, allowing the test to be re-run on new markets as models improve without becoming stale.
Immutable Archive: A leakage-free retrieval backend built from 25 months of public Reddit snapshots (Jan 2024–Jan 2026), indexed for dense semantic search and by-ID lookup, with strict temporal filtering.
Dual Reference Predictors: A unified interface supporting a zero-shot baseline and a retrieval-grounded agent, isolating the contribution of evidence retrieval.
Empirical Analysis: A study identifying specific domains where retrieval improves forecasting and where it degrades performance due to speculative noise.

4. Experimental Results

The authors benchmarked 9 open-weight LLMs (including Qwen3, Llama 3, Gemma 3, and Ministral families) on a balanced set of 216 resolved Polymarket markets.

Overall Performance: Retrieval lowered the Brier score for 8 out of 9 models. The most significant improvement was seen in Qwen3-32B, which reduced its Brier score from 0.234 to 0.179 (a 23% drop).
Topic-Dependent Variance: The benefit of retrieval is not uniform; it depends heavily on the nature of the discussion in the pre- $t_0$ $t_{0}$ corpus:
- Retrieval Helps: In Sports, Awards, and Trading/Commodities, retrieval significantly improved accuracy. In these domains, Reddit discussions contained concrete, fact-anchored pre-event signals (e.g., team performance, price trends) that models could leverage.
- Retrieval Hurts: In Entertainment, Elections/Politics, and Tech/AI, retrieval often degraded performance. In these domains, the pre-event corpus was dominated by speculative hype and opinion (e.g., fan excitement for a song that never charted). The retrieval agent over-indexed on this "loudest" but least predictive chatter, leading to confident but incorrect forecasts.
Calibration vs. Accuracy: In some cases (e.g., Llama-3.1-8B), retrieval reduced accuracy (more wrong decisions) but improved calibration (probabilities moved closer to 0.5 rather than extreme wrong values), lowering the Brier score.
Lookback Sensitivity: Varying the cutoff $t_0$ (from 25% to 100% of the market's lifespan) showed that retrieval gains saturate well before the full available lookback, suggesting the benefits are driven by early signal rather than late-breaking news.

5. Significance and Claims

The paper argues that retrieval-grounded reasoning is not universally beneficial; its efficacy is strictly bounded by the quality of the evidence available in the pre-event information environment.

Closing the Leak: HINDCAST provides a rigorous way to evaluate foresight by ensuring the model cannot "see the future" through its training data or retrieval tools.
Signal vs. Noise: The study demonstrates that retrieval helps only when the archive carries pre-event signal (facts, data, concrete analysis). When the archive carries only speculation, retrieval can actively harm forecasting by amplifying noise.
Dynamic Evaluation: Unlike static benchmarks that decay as models are retrained, HINDCAST allows for continuous evaluation. New models can be tested against the same frozen historical archive, ensuring the test remains a measure of foresight rather than memory.

The authors conclude that while retrieval improves forecasting in fact-heavy domains, it requires careful calibration in speculative domains to avoid overfitting to pre-event hype. They frame HINDCAST as a research testbed rather than a ready-to-deploy trading tool, cautioning that uncalibrated retrieval agents can lend false authority to automated decisions.