Technical Summary: WorldCupArena

Problem Statement

Current benchmarks for Large Language Models (LLMs) and deep-research agents predominantly rely on static datasets where answers are already known, failing to evaluate a model's ability to make useful predictions before an event occurs. While recent dynamic benchmarks update questions or wait for future outcomes, they often focus on single answers or coarse event outcomes. There is a lack of evaluation frameworks that test whether models can gather current evidence and apply it consistently across a spectrum of related predictions—from match results to detailed statistics and competition outcomes. Football forecasting provides a practical setting for this challenge due to its public schedule, evolving pre-match information, and rich final records containing scores, lineups, events, and tactical statistics.

Methodology

WorldCupArena is a dynamic benchmark designed to evaluate LLMs and deep-research agents on the 2026 FIFA World Cup. The benchmark operates on a strict temporal protocol:

1. Pre-Match Prediction: 24 hours before kickoff, models are evaluated under two evidence settings:
   • S1 (Standardized Context): Models receive a fixed package of squads, recent form, statistics, news, and odds.
   • S2 (Self-Directed Search): Agents receive only the fixture and must search for evidence themselves.
2. Prediction Scope: Models must output a JSON structure containing:
   • Match Level: Result probabilities, predicted score, starting lineups, formations, scorers, events (goals, cards, substitutions), and match statistics (possession, shots, etc.).
   • Competition Level: Group tables, knockout pairings, and the tournament champion.
   • Rationale: A written explanation justifying the predictions.
3. Evaluation: Predictions are scored against official match records after the event. The evaluation is structured into five layers (Figure 1):
   • T1: Result & Score (Winner, exact score, scoreline closeness).
   • T2: Players & Lineups (Starters, formation, scorers, assists).
   • T3: Events & Timeline (Goal times, substitutions, cards).
   • T4: Tactics & Statistics (Possession, shots, corners).
   • T5: Competition Outcome (Group order, bracket, champion).

Metrics and Scoring:

• Result Accuracy: Percentage of correct winner predictions (Home/Draw/Away).
• Exact-Score Accuracy: Percentage of matches with the exact goal totals predicted.
• Scoreline Score: A metric rewarding "close misses" (e.g., predicting 2-1 for a 3-1 result) by combining correct winner, goal difference, and total goal proximity.
• Composite Score: A weighted sum of all layers (T1 weighted at 0.40, T5 at 0.10).
• Calibration: Raw scores are transformed using a fixed S-shaped function to improve leaderboard visibility without altering rankings. Missing official data fields are excluded from scoring rather than penalized as errors.

Key Contributions

1. Reusable Dynamic Benchmark: The authors constructed a benchmark capable of evaluating models on future football competitions without relying on known outcomes, demonstrated here on all 104 matches of the 2026 World Cup.
2. Granular Evaluation Framework: Unlike traditional forecasting that focuses solely on results, this benchmark evaluates detailed predictions (players, events, tactics) and competition trajectories simultaneously, revealing that models with similar result accuracy can differ significantly in detail.
3. Evidence Comparison: The study compares models given static evidence against those performing self-directed web searches, while maintaining a consistent prediction format for both.
4. Open Science: The paper publishes the code, prompts, predictions, and evaluation scripts, allowing for reproducibility and future extension to other leagues.

Results

The evaluation covered 13 systems (including Claude Opus 4.7, GPT-5.4, GLM-5.1, Kimi, and others) and compared them against betting-market (Polymarket, BetVictor) and human-fan baselines.

• Result Accuracy vs. Detail: Models achieved result accuracies between 58.6% and 70.7%. However, result accuracy alone failed to distinguish performance; models with similar win-prediction rates showed sharp differences in scoreline, player, and event predictions.
• Search Performance: Adding web search (S2) did not consistently improve forecasting. For models tested in both settings, the non-search versions often scored higher on shared matches.
• Competition Prediction: Four systems correctly predicted Spain as the champion. However, only two of these (both Claude configurations) also correctly predicted the exact final pairing (Spain vs. Argentina).
• Baseline Comparison: The best system (Claude Opus 4.7 with Search) showed only marginal gains over human-fan and betting baselines in strict result accuracy (e.g., +1.0% over human fans). The most significant gain was in the Scoreline metric, indicating models are better at predicting scores close to the actual result rather than exact scores.
• In-Play Forecasting: In a separate track, models updated predictions during matches. Accuracy improved significantly after observed score changes, suggesting models react well to new state information but do not necessarily anticipate goals independently.
• Failure Modes: Systems often failed together on upsets (e.g., all 13 systems predicted Spain to win 3-0 or 4-0 against Cape Verde, which ended 0-0), indicating a shared bias toward favorites.

Significance

The paper argues that WorldCupArena demonstrates that "result accuracy alone hides important differences." By evaluating the full spectrum of a match—from lineups to the final tournament bracket—the benchmark reveals that current models struggle with exact score prediction (accuracy < 18%) and often fail collectively on upsets.

The significance lies in shifting the evaluation paradigm from static knowledge retrieval to dynamic, pre-event forecasting. The benchmark proves that while general-purpose models can approach human and market baselines in predicting winners, they offer clearer value in generating plausible, detailed narratives (scorelines, events) even when the exact outcome is missed. The authors emphasize that the 2026 World Cup is merely the first iteration; the framework is designed to evolve with future leagues, providing a continuous, real-world testbed for models released after the tournament concludes.