🤖 machine learning

Reinforcement Learning for LLM-based Event Forecasting

This paper demonstrates that applying Group Relative Policy Optimization (GRPO) to fine-tune small LLMs (1.5B–14B parameters) with access to real-time information enables them to outperform larger models like Claude Sonnet 3.5 in forecasting future events beyond their knowledge cutoff, while also analyzing scaling capabilities and the nature of judgmental forecasting under uncertainty.

Original authors: Amit Arnold Levy

Published 2026-06-16

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Amit Arnold Levy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Teaching AI to Guess the Future

Imagine you have a very smart student (an AI) who has read almost every book in the library up to a certain date, but they haven't seen today's newspaper. You ask them to predict things that haven't happened yet, like "Will it rain tomorrow?" or "Who will win the election?"

The problem is that the student doesn't know the answer, and even if they guess perfectly, they might get it wrong just because of bad luck (like a coin flip). This paper is about teaching this student how to become a better guesser using a special training method called Reinforcement Learning, specifically a technique called GRPO.

The author, Amit Arnold Levy, found a way to train a relatively small AI model (1.5 billion "brain cells") to become a better forecaster than a much larger, more expensive AI model (like Claude Sonnet 3.5), but only under specific conditions.

1. The Problem: The "Coin Flip" Trap

In math or coding, there is usually one right answer. If you solve an equation, it's either right or wrong. But in forecasting, things are messy.

The Analogy: Imagine you are betting on a coin flip.

The Truth: The coin is fair. The correct prediction is "50% chance of Heads."
The Reality: You flip the coin, and it lands on Heads.

If you train a student by saying, "You were right if you said Heads, wrong if you said Tails," the student gets confused. Even if they correctly predicted "50%," they might get punished because the coin landed on Tails that one time. This is called Aleatoric Uncertainty (randomness you can't control).

The Paper's Discovery: The author tested this and found that if you train the AI using only the final result (Heads or Tails) as the "grade," the AI learns poorly. It's like trying to learn to play poker by only looking at who won the hand, without understanding the odds. The "noise" of the random coin flip messes up the learning.

The Solution: Instead of grading the AI on whether the coin landed on Heads or Tails, the author graded the AI on how close its percentage guess was to the "market's" guess. Think of the market as a giant pool of thousands of people betting on the coin. If the market says there is a 50% chance, and the AI says 50%, the AI gets a good grade, even if the coin lands on Tails.

2. The Training Method: "Group Relative Policy Optimization" (GRPO)

How did they teach the AI? They used a method called GRPO.

The Analogy: Imagine a teacher giving a quiz to a class of 8 students.

Old Way (PPO): The teacher keeps a separate "judge" robot in their head to decide if an answer is good. This takes up a lot of mental space (computer memory).
GRPO Way: The teacher asks all 8 students to answer the same question. Then, the teacher looks at the group. If 7 students give a bad answer and 1 student gives a great answer, the teacher says, "Okay, the one who got it right is the winner. Let's all try to think like that winner next time."

This method is efficient because the teacher doesn't need a separate judge robot; they just compare the students to each other. This saves a lot of computer memory, allowing the author to train the AI on a single powerful computer chip (an H100) in just a few hours.

3. Giving the AI "Eyes" (Context)

The AI can't predict the future if it doesn't know what's happening now. It needs to read the news.

The author gave the AI two ways to get information:

The News Summarizer: The AI asks another AI (Perplexity's Sonar) to read the internet and write a short summary of relevant news.
The Time-Traveling Librarian (Wikipedia Tool): The AI is allowed to look at Wikipedia articles, but it has to look at them as they existed before the event happened. This prevents the AI from "cheating" by reading the answer key.

The Result: The "News Summarizer" worked best. The AI learned to read the summary and make a prediction.

4. The Big Surprise: Small vs. Big

Usually, in AI, bigger is better. A giant brain (14 billion parameters) usually beats a small brain (1.5 billion parameters).

The Analogy: Imagine a small, agile race car vs. a massive, slow-moving truck.

On an Open Highway (Lots of Info): The truck (big AI) wins because it can carry more data.
On a Narrow, Foggy Path (Limited Info): The small race car (small AI) wins.

The author found that when the AI was given only a short summary of news (the "narrow path"), the small, trained AI (Qwen 1.5B) actually beat the much larger, untrained AI (Claude Sonnet 3.5).

Why? Because the big AI was confused by the limited information, while the small AI was specifically trained to be a master of that specific type of guessing game. However, if you give the AI a lot of information, the big AI catches up again.

5. What Actually Worked (and What Didn't)

What Worked: Using "Market Prices" (what people are betting on) as the "correct answer" to train the AI. This acted as a reliable ground truth.
What Failed: Trying to train the AI using the actual outcome of events (e.g., "The coin landed on Heads, so you must have predicted Heads"). This was too noisy and didn't help the AI learn the probability, just the luck.
The Limit: The AI is only as good as the news summary it reads. If the news summary lies or is wrong, the AI will make a confident but wrong prediction. The AI cannot fix bad information.

Summary

This paper is a recipe for turning a small, cheap AI into a surprisingly good fortune teller.

Don't train it on the final result of random events (like coin flips); train it on the odds (the probability).
Do use a smart group-training method (GRPO) that saves memory.
Do give it a short, summarized news feed.
Result: A small AI, trained for a few hours, can out-guess a much larger, more expensive AI when the information available is limited.

The author concludes that while AI can learn to forecast, it hits a wall if the information it's given isn't good enough. It's not magic; it's just very efficient pattern matching based on the best data available.

Technical Summary: Reinforcement Learning for LLM-based Event Forecasting

Problem Statement

The paper addresses the challenge of training Large Language Models (LLMs) to perform judgmental forecasting—predicting the probability of future binary events (e.g., election outcomes, geopolitical conflicts) that lack immediate ground truth. Unlike verifiable domains such as mathematics or coding, forecasting involves aleatoric uncertainty (inherent randomness), where the "ground truth" of a single event instance (e.g., a coin flip resulting in heads) does not necessarily reflect the true underlying probability (e.g., 50%).

The core difficulty lies in the training signal: if an LLM is trained to predict the probability of an event but is rewarded based on the binary outcome (0 or 1), the reward becomes noisy. A model predicting the correct probability (e.g., 0.5) may be penalized if the event resolves as 0, while a model predicting a deterministic 1.0 might be rewarded by chance. This paper investigates whether Reinforcement Learning (RL) can effectively train LLMs to forecast accurately despite this noise, and whether smaller models can outperform larger, more knowledgeable models when equipped with specific training methodologies and context tools.

Methodology

1. Theoretical Foundation: Aleatoric Uncertainty

The authors first establish a baseline experiment using the GSM8K dataset to test if RL can learn in the presence of aleatoric uncertainty. They define three training modes:

Regular: The model predicts a number (25–75) and is rewarded for correctness (simulating a scenario where the probability itself is the ground truth).
Randomized: The model's predicted number is treated as a probability for a Bernoulli trial. The ground truth is collapsed to 0 or 1 based on a random draw. This simulates real-world forecasting where only the outcome is known, not the true probability.
Format Only: The model receives no correctness reward, only a format reward.

Results from this phase indicated that training on the "Randomized" mode (outcome-based) is highly sensitive to the specific sequence of random outcomes ("coin flips"). In most cases, the noise from aleatoric uncertainty actively harms learning compared to having the true probability as the ground truth.

2. Training Pipeline: GRPO and Ground Truth Approximation

The primary methodology employs Group Relative Policy Optimization (GRPO), a memory-efficient RL algorithm that does not require a separate value model (critic).

Ground Truth Proxy: Instead of using binary event resolutions (which suffer from aleatoric noise), the authors use market prices from Polymarket as the ground truth. They assume that high-volume prediction market prices approximate the true aleatoric probability of an event given available information.
Context Provision: To overcome knowledge cutoffs, the LLMs are provided with context via two methods:
1. Summarized News: Using Perplexity's Sonar Reasoning Pro to generate relevant summaries of current events.
2. Wikipedia Revisions Tool: Allowing the model to query historical Wikipedia articles at specific timestamps (used for resolved events to prevent temporal leakage).
Reward Function: The models are fine-tuned using Negative Cross Entropy (or negative quadratic error) between the model's predicted probability and the Polymarket market price.

3. Experimental Setup

Models: The study fine-tunes Qwen 2.5 models ranging from 1.5B to 14B parameters.
Dataset: 512 high-volume binary questions from Polymarket.
Comparison: Results are benchmarked against Claude 3.5 Sonnet, a larger model with a later knowledge cutoff.

Key Contributions

Demonstration of GRPO for Forecasting: The paper successfully applies GRPO to train LLMs on judgmental forecasting tasks, showing that RL can significantly improve forecasting performance even in the presence of aleatoric uncertainty, provided a reliable proxy for ground truth (market prices) is used.
Small Model Superiority in Limited Context: A counter-intuitive finding is that a 1.5B parameter model (Qwen 2.5 1.5B), after GRPO fine-tuning, achieves forecasting performance superior to Claude 3.5 Sonnet on the same dataset. This is measured by the negative cross-entropy against market probabilities.
Analysis of Scaling and Bottlenecks: The study reveals that in "low information" settings (limited context summaries), model size does not yield a scaling advantage post-training. All models plateau at similar performance levels, suggesting that the quality and quantity of provided context (the summaries) act as the primary bottleneck, not the model's internal capacity.
Taxonomy of Uncertainty: The work clarifies the distinction between epistemic uncertainty (lack of knowledge, reducible) and aleatoric uncertainty (inherent randomness, irreducible), demonstrating that training on binary outcomes (aleatoric) is significantly less efficient than training on probability distributions (epistemic).

Results

Performance Gains: GRPO training improved the initial forecasting capability of Qwen models. For instance, the Qwen 1.5B model improved its Mean Negative Cross-Entropy (NCE) from -1.66 to -0.415.
Comparison with Larger Models: The fine-tuned 1.5B model achieved an NCE of -0.415, outperforming the baseline Claude 3.5 Sonnet, which scored -0.62 on the same validation set.
Scaling Limits: Increasing model size from 1.5B to 7B did not significantly improve the final performance plateau in the low-information setting. All Qwen variants converged to similar NCE scores (approx. -0.3 to -0.4), indicating that the information provided by the summarization tool limited further gains.
Failure Modes: The paper notes that models consistently fail when the summarization tool provides incorrect information. In a specific example regarding an Israel-Iran ceasefire, the summarizer incorrectly claimed the market had already resolved, leading the model to predict a 100% probability for an event that the market priced at 0%.

Significance and Claims

The paper claims that judgmental forecasting is a tractable domain for RL, distinct from the perfectly verifiable domains of math and code. Its primary significance lies in demonstrating that:

Efficiency: High-performance forecasting can be achieved with small models (1.5B) using GRPO, which is computationally cheaper and faster to train than larger models, provided the training signal (market prices) is accurate.
Data over Size: In forecasting tasks with limited context, the quality of the retrieved information and the training objective (using market probabilities vs. binary outcomes) are more critical determinants of success than model scale.
Practical Feasibility: The training pipeline is feasible on limited hardware (e.g., a single H100 for 2-3 hours for models $\le$ 7B), making advanced forecasting capabilities accessible without massive compute clusters.

The authors conclude that while larger models generally outperform smaller ones out-of-the-box, strategic fine-tuning with GRPO and high-quality ground truth proxies can reverse this hierarchy in specific, information-constrained regimes. They also highlight that future work must address the bottleneck of information retrieval, as current performance is capped by the accuracy of the context summaries provided to the model.