Automating Forecasting Question Generation and Resolution for AI Evaluation

This paper presents an automated system using LLM-powered web research agents to generate and resolve diverse, real-world forecasting questions at scale, demonstrating high-quality question creation and resolution rates that surpass human-curated platforms while effectively evaluating and improving AI forecasting performance.

The Gist

Imagine you are trying to teach a group of very smart robots how to predict the future. You want to know if they are getting smarter, just like you might test a student by giving them harder math problems.

But here's the catch: You can't just ask them random questions. If you ask, "Will it rain tomorrow?" the answer is too easy. If you ask, "Will the stock market crash?" the answer is too vague. To really test a robot's "brain," you need thousands of specific, tricky questions about the real world, and you need to know the answers later to grade them.

In the past, humans had to write all these questions by hand. It was slow, boring, and expensive. Or, researchers used computer-generated questions about things that happen every day (like the weather), which wasn't very interesting or useful.

This paper is about a team that built a robot factory to solve this problem. They created an automated system that writes, checks, and grades its own future-prediction questions.

Here is how their "Robot Factory" works, broken down into simple steps:

1. The Seed Planters (Finding Inspiration)

Instead of staring at a blank page, the system starts with "seeds." Imagine these are clippings from news articles, stock market reports, or government announcements.

• The Analogy: Think of these seeds like planting a garden. You don't just plant "a flower"; you plant a specific seed (like a news story about a new law) and see what kind of question grows from it.

2. The Drafters (The "What If?" Agents)

The system takes those news seeds and asks a team of AI agents (robots with internet access) to turn them into "proto-questions."

• The Analogy: These are like rough drafts written by a creative writer. They might say, "Will the EU pass a new law?" This is a good idea, but it's too vague. Which law? When? How do we know if it passed?

3. The Editors (The "Refiners")

Another team of AI agents takes those rough drafts and polishes them. They add strict rules so there is no confusion later.

• The Analogy: Imagine a strict editor who says, "Okay, 'Will the EU pass a law?' is too vague. Let's change it to: 'Will the EU publish a specific regulation on recycled plastic in their official journal by December 31st?'" Now, the question is clear, and we know exactly where to look for the answer.

4. The Quality Control Inspectors (The Verifiers)

Before the questions go out to the robots, a panel of "inspectors" checks them. They ask:

• Is this question too easy? (If the answer is obvious, it's a bad test.)
• Is it too hard to find the answer? (If no one can find the data, the test is broken.)
• Is it ambiguous? (Could two people argue about the answer?)
• The Analogy: This is like a teacher grading a test before giving it to students. If a question is confusing or the answer key is missing, the teacher throws it out.

5. The Final Product

The system generated 1,499 high-quality questions covering topics like politics, space launches, weather, and court cases.

• The Result: The system was so good that it produced questions that were just as good (or better) than those written by human experts on famous forecasting websites. In fact, it made fewer mistakes than humans did!

6. The Exam (Testing the Robots)

Once the questions were ready, the team tested different AI models (like GPT-5, Gemini 3 Pro, etc.) to see how well they could predict the answers.

• The Finding: The smarter the robot, the better it did. The newest, most powerful models got the highest scores.
• The "Super-Strategy": They also found that if they broke a big question down into smaller, easier questions (like solving a puzzle piece by piece), the robots got even smarter and more accurate.

Why Does This Matter?

Think of this system as a gym for AI brains.

• Before, we didn't have enough good "weights" (questions) to lift.
• Now, we have a machine that can generate an endless supply of heavy, diverse, and challenging weights.
• This allows us to see exactly how smart our AI is getting and helps us build better tools for making real-world decisions, from business investments to government policy.

In a nutshell: The authors built a self-running machine that invents its own tricky future-prediction puzzles, checks to make sure they are fair, and then uses them to prove that smarter AI models are indeed getting better at guessing the future. It's a massive leap forward in testing how "intelligent" our computers are becoming.

Technical Summary

Here is a detailed technical summary of the paper "Automating Forecasting Question Generation and Resolution for AI Evaluation."

1. Problem Statement

Evaluating Large Language Models (LLMs) on forecasting tasks is critical for assessing progress toward Artificial General Intelligence (AGI), as forecasting is a robust, ungameable measure of general intelligence. However, creating high-quality evaluation datasets faces two major bottlenecks:

• Sample Complexity: Unlike classification, forecasting requires observing outcomes over many diverse questions to distinguish between well-calibrated and poorly-calibrated models.
• Data Scarcity & Cost: High-quality forecasting questions require precise, unambiguous resolution criteria and must be grounded in real-world events. Existing benchmarks rely on labor-intensive human curation (e.g., Metaculus) or automated generation from static/recurring data sources (e.g., stock prices, weather), which limits diversity, relevance, and statistical power.

The authors aim to solve this by developing a system that can automatically generate, refine, and resolve a large-scale, diverse set of real-world forecasting questions with human-level or superior quality.

2. Methodology

The authors propose an agentic workflow powered by LLMs and web-research agents to automate the entire lifecycle of forecasting questions. The system operates in a multi-stage pipeline (Figure 1):

A. Question Generation Pipeline

1. Seeding: The process begins with "seeds" derived from real-world data sources (GDELT news, Media Cloud, and Stockfisher revenue forecasts) to ground questions in current events.
2. Proto-Question Generation: A ReAct-style agent (using web search tools) generates 1–7 "proto-questions" per seed. These are rough ideas about future events but lack precise resolution criteria.
3. Refinement: A second ReAct agent refines these into fully operationalized questions. It adds precise, objective resolution criteria, defines all terms, and ensures the question can be resolved by a specific, verifiable source by a fixed date.
4. Verification (Filtering): A suite of specialized verifier agents evaluates each refined question against four criteria:
   • Quality: Is the task non-trivial and does it reward research?
   • Ambiguity: Are the resolution criteria clear and objective?
   • Resolvability: Can an AI agent autonomously resolve the question?
   • Triviality Check: A simple forecaster checks if the answer is obvious (e.g., >98% or <2% probability).
5. Deduplication: Embeddings (text-embedding-3-large) and DBSCAN clustering, followed by LLM verification, remove duplicate or near-duplicate questions.

B. Question Resolution

To evaluate the questions, the system resolves them using an ensemble of agents:

• Three agents powered by Gemini 3 Pro (two with identical prompts, one with a different prompt) attempt to resolve the question.
• If they disagree, a tiebreaker agent powered by Opus 4.5 is invoked.
• The system aims for high accuracy in determining the binary outcome (Yes/No) or flagging a question for annulment.

C. Evaluation Logistics

• Dataset: 2,500 seeds generated 14,073 proto-questions, which were refined and filtered down to 1,499 unique, high-quality questions.
• Timeline: Questions were generated in late 2025, forecasted in late 2025, and resolved in early 2026.
• Forecasting Models: Various models (GPT-5, Gemini 3 Pro, etc.) were tested using a two-stage process: (1) Web research to gather evidence, and (2) A single LLM call to produce a probability forecast based on that evidence.

3. Key Contributions

• First Automated Web-Research Generation System: Unlike previous benchmarks relying on static data, this system uses flexible web-research agents to generate timely, grounded questions from the live internet.
• High-Quality Automated Curation: The system achieves a ~96% rate of producing verifiable, unambiguous questions, exceeding the quality rate of human-curated platforms like Metaculus.
• Automated Resolution with High Accuracy: The automated resolver achieves ~95% accuracy compared to human expert judgments, with an estimated annulment rate of 3.9% (compared to Metaculus' historical ~8%).
• Demonstrated Scalability: Successfully generated a dataset of 1,499 diverse questions covering geopolitics, economics, policy, and science, a scale difficult to achieve with human curation alone.

4. Results

The paper validates the system through multiple metrics:

• Diversity & Consequentiality: Topic clustering shows the questions span 12 major domains (e.g., US Policy, International Security, Macroeconomics). Intra-cluster analysis confirms low similarity (mean score 1.32/4), indicating genuine diversity rather than trivial variations.
• Expert Validation: External expert forecasters rated 75.2% of the generated questions as "accept" (high quality), with only 8.1% rated as "hard reject" (ambiguous/flawed).
• Model Performance Correlation: The benchmark successfully discriminates between model capabilities.
  • Gemini 3 Pro achieved the best Brier score (0.134).
  • GPT-5 scored 0.149.
  • Gemini 2.5 Flash scored 0.179.
  • Observation: Larger and newer models consistently outperformed smaller/older counterparts, proving the benchmark rewards intelligence.
• Impact of Effort (Subquestion Decomposition): When a "subquestion decomposition" strategy was applied (breaking a question into 3–5 sub-questions, researching them, and aggregating), the Brier score improved significantly from 0.141 to 0.132. This demonstrates the benchmark rewards increased research effort and reasoning depth.

5. Significance

This work represents a paradigm shift in AI evaluation for forecasting:

1. Solving the Data Bottleneck: It provides a scalable, automated solution to the "sample-hungry" nature of forecasting evaluation, enabling the creation of massive, diverse datasets without prohibitive human cost.
2. Benchmarking AGI Progress: By creating a robust, ungameable benchmark that correlates with model intelligence and effort, the system offers a reliable metric for tracking progress toward AGI.
3. Practical Utility: The system's ability to generate consequential, real-world questions (e.g., policy outcomes, market shifts) suggests it can be used not just for research, but to directly improve forecasting capabilities in decision-making contexts.
4. Validation of Agentic Workflows: The success of the multi-agent verification and resolution pipeline validates the use of LLM agents for complex, multi-step tasks requiring research, reasoning, and quality control.

In conclusion, the authors have demonstrated that LLM-powered agents can autonomously generate and resolve forecasting questions at a scale and quality that rivals or exceeds human curation, providing a critical infrastructure for the next generation of AI evaluation.