💬 NLP

WorldCup Arena: Prospective, Leakage-Free Evaluation of Frontier LLMs on a Live Tournament

This paper presents a leakage-free evaluation of six frontier large language models during the 2026 FIFA World Cup, revealing that while the models achieve bookmaker-level accuracy on match outcomes, they exhibit shared limitations such as over-reliance on favorites, poor differentiation among themselves, and a tendency to under-predict draws and specific scorelines.

Original authors: Zhenran Wang, Zhonghan Bian, Jinsong Li, Zhangyang Qi

Published 2026-08-05

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhenran Wang, Zhonghan Bian, Jinsong Li, Zhangyang Qi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but the crime hasn't happened yet. In the world of artificial intelligence, we often test "smart" computer programs by asking them questions about things that have already occurred, like "Who won the 2022 World Cup?" The problem is that these programs might have just memorized the answer from the internet, rather than actually figuring it out. To truly test if a brain is smart, you need to ask it to predict the future—something that doesn't exist yet and can't be Googled. This is the frontier of "forecasting," where we see if a machine can reason its way to an answer that no one knows yet.

This paper, titled WORLDCUP ARENA, takes that idea and runs a massive, real-time experiment. Instead of asking a computer to guess the winner of a game that finished years ago, the researchers set up a live tournament: the 2026 FIFA World Cup. They invited six of the world's most advanced AI models to act as sports analysts. Every day, before a single match kicked off, these AIs had to look at the teams, the weather, the referees, and the latest news, and then make seven different predictions for the game (like who will win, how many goals will be scored, and the exact score). The catch? The games hadn't happened yet. The answers didn't exist anywhere on the internet. This meant the AIs couldn't access the score; they had to actually think.

The researchers wanted to see if these super-smart computers could beat the odds, or if they would just follow the crowd. They found that while the AIs were impressive, they weren't magic. On average, they got about 63.9% of the match outcomes right. That sounds good, but it's almost exactly the same as just blindly betting on the team the bookmakers thought would win. In fact, the AIs were so cautious that they rarely predicted a tie, even though ties happened often, and they kept guessing the same boring score (2–1) over and over again.

Here is the most surprising part: the six different AI models were almost identical in their thinking. They agreed with each other 76% of the time. If you asked them to vote on the winner, the "majority vote" didn't do any better than the best single AI. It turns out that even the most advanced "thinking" models of 2026 are all following the same playbook: they are very good at looking at the big picture (like predicting which country wins the whole tournament) but struggle with the tiny, messy details of a single, close game. The paper concludes that right now, these AI systems aren't sharply different from one another; they are all just very good at mimicking human betting patterns, but they haven't figured out how to truly "see" the future any better than a human expert with a calculator.

The Experiment: A Live Tournament of Minds

The researchers set up a "leakage-free" arena. In normal tests, scientists have to worry that the AI might have "leaked" the answer from its training data. But here, because the matches were happening in real-time in 2026, the answers literally didn't exist when the questions were asked. The setup was rigorous:

The Players: Six frontier AI models (Claude, GPT, Gemini, Kimi, GLM, and Seed).
The Rules: Each model had to use its "extended thinking" mode (the deepest level of reasoning available) and its own built-in web search tool to read the latest news, injury reports, and odds.
The Task: For every one of the 104 matches in the tournament, the AI had to fill out a "prediction card" with seven markets: who wins, the exact score, total goals, and more.
The Data: The AIs were given a "dossier" for each team that updated daily, growing from a small summary to a massive file of over 1 million characters by the end of the tournament.

What They Found: The "Herd" Mentality

The results painted a clear, slightly funny picture of how these AIs think.

1. They Follow the Crowd, Not the Crystal Ball
The AIs didn't outsmart the bookmakers. They simply sided with the bookmakers' favorite team 85.6% to 90.4% of the time. Their average accuracy of 63.9% was statistically the same as just mechanically backing the favorite (which gets 64.4%). They weren't discovering new insights; they were echoing the market.

2. The "Draw Blindness"
The AIs were terrible at predicting ties. In the tournament, 27 matches ended in a draw. The models only predicted 8 to 14 draws. They were so afraid of the tie that they barely considered it, even though it happens frequently in football.

3. The "2–1" Obsession
When guessing the exact score, the models got stuck in a rut. They placed 28.0% of all their score predictions on a single result: 2–1. They ignored the vast variety of possible scores and crowded their guesses onto one "prototypical" result.

4. Big Picture vs. Small Details
The models were "macro-strong but micro-weak."

Macro: They did great at predicting the tournament winner and group standings.
Micro: They fell apart in the closest games. In the Round of 32 (where teams were well-matched), they got 86.5% right. But in the Semi-Finals, where the dossiers were richest with information and the teams were evenly matched, their accuracy collapsed to 8.3%. The more they knew about the teams, the worse they did at picking the winner of a tight game.

5. The "Herd" Effect
The six models agreed with each other 76.0% of the time. Because they were all thinking so similarly, asking them to vote didn't help. A "majority vote" of the six models added nothing to the accuracy. They were all stuck in the same loop.

The Leaderboard: A Tight Race

The final scores were incredibly close, showing that the current generation of AI isn't sharply differentiated.

Claude took first place with 897 points.
GLM came in last with 813 points.
The gap between first and last was only 84 points (about 10.3%).

When the researchers ran statistical tests (called "bootstrap intervals") to see if the winner was truly better, the results were shaky. Claude was the winner in 53.6% of the simulated re-runs, but the difference between the top models was often within the margin of error. For example, the difference between the first and second place was so small that if you ran the tournament again, the order might flip.

The Takeaway

This study suggests that while today's most advanced AI models are powerful tools for gathering information and reasoning, they haven't yet learned to break free from the patterns of the past to truly predict the future. They are excellent at summarizing what we know, but when faced with a truly uncertain event where the answer doesn't exist yet, they tend to play it safe, follow the crowd, and guess the most "average" outcome.

The paper concludes that the current generation of frontier systems is not sharply different from one another. They all share the same strengths (big-picture thinking) and the same weaknesses (fear of draws, obsession with average scores). The "magic" of AI forecasting, it seems, is still just a very sophisticated version of following the betting odds.

Technical Summary: WORLDCUP ARENA

Problem Statement
Current benchmarks for evaluating the forecasting abilities of Large Language Models (LLMs) are predominantly retrospective. In these settings, the event has already occurred, and the answer exists within the model's training data or on the public web. Consequently, evaluations must rely on complex filtering or anonymization to prevent data leakage (memorization), which introduces uncertainty about whether a model is truly reasoning or merely recalling. There is a lack of evaluation protocols where the answer genuinely does not exist at the time of the query, making "contamination" a category error rather than a mitigatable threat.

Methodology
The authors present WORLDCUP ARENA, a prospective, leakage-free evaluation conducted over the 39 days of the 2026 FIFA World Cup (June 10 – July 19, 2026). The study involved six frontier LLMs (Claude, GPT, Gemini, Kimi, GLM, and Seed), all configured with their highest available "thinking" settings and native server-side web search capabilities.

Task Design: Before every kickoff, models were tasked with filling out a seven-market prediction card for the upcoming match. The markets included: Match Outcome (1X2), Handicap, Half-time/Full-time, Over/Under 2.5 goals, Both Teams to Score, Correct Score, and Odd/Even total goals. Additionally, models predicted 12 group winners and a pre-tournament outright pool (champion, finalists, etc.).
Input Data: Models received a "rolling dossier" for each team, updated daily across 46 timestamped snapshots. These dossiers included FIFA rankings, squad data, recent results, and news. Crucially, the input for match $t+1$ included the results of match $t$ but nothing beyond that.
Leakage Controls: The evaluation is "leakage-free by construction" via three procedural locks:
1. Snapshot Lock: Models only received data current to the moment of the query.
2. Kickoff Lock: Predictions were strictly gated to occur before the match start time, preventing retrieval of live scores.
3. Template Lock: Prompts, market definitions, and scoring weights were fixed and archived verbatim before model invocation.
Scoring: A deterministic script scored 4,494 predictions based on official 90-minute results. Points were weighted across markets (e.g., Handicap worth 4 points, Match Outcome 2 points).

Key Contributions

Prospective Evaluation Protocol: A fully prospective, leakage-free framework for LLM forecasting executed end-to-end on a complete tournament with objective ground truth, eliminating the need for anonymization proxies.
Public Benchmark: The release of a frozen dataset comprising 48 team dossiers across 46 snapshots, the full 104-match fixture list with metadata, official results, and the scoring code.
Uncertainty-Aware Analysis: The application of bootstrap intervals and five alternative scoring designs to assess the robustness of model rankings.

Results

Leaderboard: The six models spanned a narrow range of 897 to 813 points. Claude finished first (897), followed by Seed (853), GPT (845), Kimi (839), Gemini (827), and GLM (813). However, bootstrap resampling (10,000 iterations) revealed that the ordering is not statistically sharp; the top model (Claude) has only a 53.6% probability of being first in a resampled tournament, and the middle of the pack frequently reorders.
Performance vs. Baselines:
- Market Tracking: Models averaged 63.9% accuracy on match outcomes, which is statistically indistinguishable from mechanically backing the bookmaker's favorite (64.4%). They generally tracked the betting market rather than beating it.
- Conservatism: Models exhibited specific behavioral biases: they under-committed to draws (predicting 8–14 when 27 occurred) and goals, and showed "mode collapse" by clustering correct score predictions heavily on a single prototypical result (2–1 accounted for 28% of picks).
- Herding: Models agreed with each other on 76.0% of match outcomes. Consequently, a majority vote offered no improvement over individual models.
- Difficulty Correlation: Accuracy was inversely correlated with the richness of information. Models performed best in the Round of 32 (86.5% accuracy) where fixtures were lopsided, but performance collapsed in the semi-finals (8.3% accuracy) where dossiers were richest and matches were tightest.
- Macro vs. Micro: Systems performed significantly better on tournament-level questions (macro) than on individual match predictions (micro).

Significance and Claims
The paper claims that the current generation of frontier LLMs is not sharply differentiated on this specific forecasting task. The study establishes that these systems share a set of behaviors: they act as conservative proxies for the betting market, fail to leverage deep information in tight matches, and exhibit high inter-model agreement that does not translate to higher accuracy.

The authors position WORLDCUP ARENA as the "control condition" for their companion work, SOCIETYBENCH. While SOCIETYBENCH uses anonymized retrospective data to test breadth across 25 events, WORLDCUP ARENA sacrifices breadth for depth and purity, proving that when the "memory assumption" is removed entirely, the models' capabilities appear more uniform and their limitations more pronounced. The paper concludes that forecasting unplayed matches sits outside the training objectives of these models, and their performance is largely a by-product of general reasoning and retrieval capabilities that converge across different architectures.

The Experiment: A Live Tournament of Minds

What They Found: The "Herd" Mentality

The Leaderboard: A Tight Race

The Takeaway

More like this