💰 quantitative finance

Foresight Arena: An On-Chain Benchmark for Evaluating AI Forecasting Agents

This paper introduces Foresight Arena, a permissionless, on-chain benchmark that evaluates AI forecasting agents on real-world prediction markets using trustless smart contracts and proper scoring rules to rigorously measure predictive accuracy while preventing overfitting and data contamination.

Original authors: Maksym Nechepurenko, Pavel Shuvalov

Published 2026-05-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Maksym Nechepurenko, Pavel Shuvalov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a giant, digital arena where AI "crystal balls" compete to see the future. This paper introduces Foresight Arena, a new way to test if Artificial Intelligence is actually good at predicting real-world events, or if it's just guessing.

Here is the breakdown of how it works, using simple analogies.

1. The Problem: Why Old Tests Were Flawed

Before this, testing AI forecasters had three big holes:

The "Cheat Sheet" Problem: Old tests used static questions (like a fixed quiz). AI models might have seen the answers while they were being trained, so they weren't really "predicting"; they were just remembering.
The "Referee" Problem: Most tests relied on a human or a central company to keep score. If that referee was biased or hacked, the results were fake.
The "Trader vs. Seer" Problem: Some tests measured how much money an AI made betting on events. But making money isn't just about being right; it's about when you bet and how much you risked. An AI could be terrible at predicting the future but still make money by being a lucky gambler.

2. The Solution: A Trustless, Digital Stadium

Foresight Arena fixes this by building the test on a blockchain (a public, unchangeable digital ledger).

The "Commit-Reveal" Game: Imagine a game of poker where you have to write your guess on a piece of paper, seal it in an envelope, and put it on the table before anyone else can see it. Only after everyone has sealed their guesses do you open the envelopes. This stops AI agents from cheating by looking at what others guessed first.
The "No-Human-Judge" Rule: The results aren't decided by a person. They are automatically read from a public, decentralized system (Polymarket/Gnosis). If the event happens, the blockchain knows. No one can change the score after the fact.
The "Free Entry" Rule: Anyone (or any AI) can join without asking for permission.

3. The Scorecard: Measuring "Truth" vs. "Luck"

Instead of counting money, the arena uses a special math formula called the Brier Score.

The Analogy: Think of it like a weather forecaster. If they say "There is a 90% chance of rain" and it rains, they get a good score. If they say 90% and it's sunny, they get a bad score. The score measures how close their confidence was to reality.
The "Alpha Score" (The Edge): This is the paper's secret sauce. It doesn't just ask, "Were you right?" It asks, "Were you more right than the crowd?"
- Imagine a crowd of 1,000 people guessing the winner of a sports game. The "Market Consensus" is the average guess of that crowd.
- If an AI guesses the same as the crowd, its score is zero.
- If the AI guesses differently and turns out to be right, it gets a positive score. This proves the AI found a piece of information the crowd missed.

4. The Experiment: Who Won?

The authors ran a live test for 50 rounds involving five top-tier AI models (from companies like Anthropic, OpenAI, Google, etc.) and a "Random Baseline" (a computer guessing numbers randomly).

The Results:

The Random Guessers: They failed miserably, proving the test works.
The Top AI Models: Three models (Claude, GPT, and Gemini) did slightly better than the crowd, but the difference was tiny. It was like a race where the top three runners finished within a fraction of a second of each other. The test wasn't long enough to say for sure who was the absolute fastest.
The "Copycat" Models: Two models (Grok and GLM) tried to guess what the crowd was thinking but added some "noise" (random errors). They actually did worse than just copying the crowd perfectly. This is a key finding: Trying to be slightly different from the crowd when you aren't actually smarter just hurts your score.

5. The "Anatomy" of a Good Prediction

The paper breaks down why an AI wins or loses using a "Murphy Decomposition" (a fancy way of taking apart a score):

Resolution (The "Sharpness"): Can the AI tell the difference between a likely event and an unlikely one? The winners were "sharper" than the crowd.
Reliability (The "Honesty"): When an AI says "70% chance," does it happen 70% of the time? The winners were very honest about their confidence.
The Lesson: To beat the market, you need to be sharper than the crowd. Just being "calm" or "honest" isn't enough if you aren't seeing things the crowd missed.

6. The Big Takeaway

This paper built a permanent, unchangeable reputation system for AI.

In the past, an AI company could claim, "Our AI is the best predictor!" and show you a spreadsheet they made up.
Now, with Foresight Arena, an AI has a public, unforgeable track record on the blockchain. You can go look at the history yourself.

The Bottom Line:
The paper concludes that while current AI models are much better than random guessing, they are currently very close to the "collective wisdom" of the human crowd. To prove an AI is truly superior, we need to keep running these tests for much longer (hundreds of rounds, not just 50) to see if the tiny differences add up to a clear winner.

What the paper does NOT claim:

It does not say these AIs can predict the stock market for profit (that's a different game).
It does not say these AIs are ready to run governments or hospitals.
It does not claim the current results are the final word; it explicitly states the test needs to run longer to separate the top performers.

1. Problem Statement

The paper identifies three fundamental limitations in current methods for evaluating AI forecasting capabilities:

Dataset Contamination: Static benchmarks (fixed questions/resolutions) are vulnerable to training-data leakage, where models appear accurate simply because they memorized the test set during pre-training.
Centralized Trust: Existing frameworks rely on central authorities to record predictions and calculate scores, creating risks of retroactive manipulation or selective reporting, which undermines commercial trust.
Improper Metrics: Evaluating agents via Trading Profit-and-Loss (PnL) conflates predictive accuracy with trading strategy (position sizing, timing, and risk appetite). An agent can lose money despite accurate predictions or make money despite poor predictions due to luck or aggressive betting.

The authors argue that a rigorous evaluation requires an environment that is resistant to overfitting, decentralized (trustless), and grounded in incentive-compatible scoring rules.

2. Methodology & System Design

Foresight Arena is a live, permissionless, on-chain benchmark deployed on Polygon PoS that evaluates AI agents on real-world binary prediction markets sourced from Polymarket.

Core Architecture

Commit-Reveal Protocol: Agents submit a cryptographic hash of their predictions (commit phase) before the deadline to prevent observing competitors' moves. They reveal the actual probabilities and a salt later (reveal phase). This ensures forecast independence.
Trustless Resolution: Outcomes are resolved automatically via the Gnosis Conditional Token Framework (CTF), the same oracle used by Polymarket. No human curator can manipulate the resolution.
Gasless Participation: Agents interact via a relayer using EIP-712 signed messages, allowing participation without holding native tokens (POL).
Persistent Reputation: Scores are accumulated on the ERC-8004 Reputation Registry, creating a tamper-proof, verifiable track record for each agent.

Scoring Framework

The system uses two primary metrics:

Brier Score: A proper scoring rule ( $S = (p - x)^2$ ) that measures the accuracy of probabilistic forecasts. It is strictly minimized only when the agent reports their true belief.
Alpha Score ( $\alpha$ ): A novel metric measuring the agent's edge over the market consensus.
- Formula: $\alpha = B_{base} - B_{agent}$ , where $B_{base}$ is the Brier score of the market mid-price.
- A positive $\alpha$ indicates the agent outperformed the collective wisdom of the market.

Statistical Analysis

The authors provide a formal mathematical treatment of the scoring rules:

Murphy Decomposition: They decompose the Brier Score into Uncertainty (UNC), Reliability (REL), and Resolution (RES).
- $\alpha$ is shown to be the sum of a Resolution Gain (better discrimination than the market) and a Reliability Gap (better calibration than the market).
Power Analysis: They derive closed-form expressions for the variance of Alpha and calculate the sample size required to detect a specific edge ( $\alpha^*$ $α^{*}$ ).
- Finding: Detecting a true edge of $\alpha^* = 0.02$ requires ~350 predictions (50 rounds of 7 markets). Detecting $\alpha^* = 0.01$ requires ~4x that amount.

3. Key Contributions

On-Chain Verifiability: The first benchmark where all predictions, commitments, and scores are recorded on a public blockchain, allowing any third party to independently replay and verify an agent's history without trusting the organizer.
Trustless Resolution: Elimination of centralized arbiters by using the Gnosis CTF for outcome resolution.
Formal Alpha Score Analysis: A rigorous statistical proof that Alpha is a proper scoring rule and a derivation of the variance and sample size requirements for reliable detection of forecasting skill.
Murphy-Decomposition Insight: Demonstrating that "beating the market" is mathematically distinct from "being calibrated," allowing for the diagnosis of specific failure modes (e.g., market-tracking with noise vs. genuine miscalibration).
Permissionless Infrastructure: An open-source, gasless system allowing any Ethereum-compatible address to participate and build persistent on-chain credentials.

4. Experimental Results

The authors conducted a 50-round live evaluation involving five frontier Large Language Model (LLM) agents and a random baseline.

Agents Evaluated: claude-opus-4-5, gpt-5-2, gemini-3-pro, grok-4-1, glm-4-7, and a Random Baseline.
Overall Performance:
- Random Baseline: Performed significantly worse than all LLMs ( $\bar{\alpha} \approx -0.14$ ), confirming LLMs possess genuine forecasting ability.
- Top Tier: claude-opus-4-5, gpt-5-2, and gemini-3-pro achieved small positive Alpha scores ( $+0.003$ to $+0.005$ ). While positive, these were not statistically significant at the 50-round horizon (consistent with power analysis predicting significance requires $\alpha^* \ge 0.02$ ).
- Underperformers: grok-4-1 and glm-4-7 achieved negative Alpha scores. The Murphy decomposition revealed they suffered from low resolution (they tracked the market price but added noise) rather than just calibration errors.
Domain Specialization:
- Crypto: LLMs showed the strongest edge here, likely due to the ability to integrate breaking news faster than the market mid-price.
- Geopolitics: All agents underperformed the market, suggesting LLMs struggle with low-frequency, high-ambiguity political events.
- Sports/Economics: Mixed results with specific models showing domain strengths.

5. Significance and Implications

Decoupling Prediction from Trading: By using proper scoring rules instead of PnL, Foresight Arena isolates the model's knowledge of the world from its trading strategy. This reveals failure modes (like "market tracking with noise") that PnL-based evaluations would miss.
Verifiable AI Credentials: The system proposes a new standard for AI evaluation: on-chain credentials. These are immutable, cumulative records of performance that cannot be falsified, essential for the commercial licensing and deployment of autonomous agents.
Statistical Honesty: The paper highlights that current short-horizon evaluations (e.g., 50 rounds) lack the power to distinguish between top-tier frontier models. It establishes that long-term, continuous evaluation is necessary to reliably rank AI forecasting capabilities.
Design Guidance: The analysis suggests that for an agent to beat the market, it must prioritize fresh information (resolution) and controlled boldness, rather than simply echoing market prices or being overconfident.

In conclusion, Foresight Arena provides the first robust, decentralized, and statistically rigorous framework for measuring AI forecasting, moving beyond static datasets and profit-based metrics to a system grounded in cryptographic verification and information theory.