💻 computer science

Diversity is the Strength of the AI Crowd

This paper demonstrates that maximizing the accuracy of AI forecasting ensembles requires strategically combining diverse models with complementary errors rather than simply aggregating multiple forecasts from similar high-performing LLMs.

Original authors: Matthew Aitchison, Scott Jeen, Toby Shevlane, Ben Day

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Matthew Aitchison, Scott Jeen, Toby Shevlane, Ben Day

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict the outcome of a series of coin flips, but instead of flipping a coin, you are asking a group of very smart, but slightly different, computers (AI models) to guess whether a future event will happen or not.

This paper asks a simple question: If you have a limited number of "guesses" you can make, should you ask your single smartest computer to guess five times, or should you ask five different computers to guess once each?

Here is the breakdown of what the researchers found, using everyday analogies.

The Old Way: "The Super-Expert"

For a long time, the standard approach was to find the single smartest AI model available and ask it the same question over and over again. The logic was: "If this model is the best, more guesses from it must be better."

The researchers call this "indiscriminate sampling." It's like asking your one best friend to write five different essays on the same topic. Even if they try to write differently, they are using the same brain, the same memories, and the same style. Their answers will be very similar to each other.

The New Discovery: "The Diverse Team"

The paper argues that this approach is wrong. Instead, the best results come from building a diverse team of different models.

Think of it like a sports team. If you have five players who are all excellent at shooting basketballs, but they all stand in the exact same spot and shoot the same way, you aren't covering the whole court. You need a mix: one player who is great at shooting, one who is great at defense, and one who is great at passing. Even if the "passer" isn't the best shooter, their unique skill makes the whole team stronger.

The Key Findings

1. Smart models think alike
The researchers tested several top-tier AI models (like GPT-5, Gemini 3 Pro, and others). They found that these "super-smart" models are actually very similar in how they think. When asked the same question, they tend to give very similar answers.

The Analogy: If you ask five identical twins the same riddle, they will likely give you the same answer. Asking them five times doesn't give you new information; it just gives you the same answer five times.

2. The "Outlier" is the MVP
One model in their test group, called Grok 4, was interesting. It wasn't the absolute #1 most accurate model on its own (it was actually #3). However, it thought very differently from the others. Its answers were less correlated with the rest of the group.

The Analogy: Imagine a puzzle. You have four pieces that fit together perfectly but leave a gap. You find a fifth piece that doesn't look like the others, but it fits perfectly into that gap. Even though the fifth piece looks "weird" compared to the others, it is the most valuable piece for finishing the puzzle.

3. Diversity beats raw accuracy
When the researchers combined the models, the winning team wasn't made of the single best model repeated five times. The winning team was a mix:

A custom-tuned model (the "specialist").
Two of the top-tier models (the "generalists").
Grok 4 (the "diverse thinker").

Even though Grok 4 wasn't the most accurate on its own, it was the hardest to replace. If you took Grok 4 out of the team, the group's performance dropped significantly. If you took out one of the "top" models, the group barely noticed.

The Big Lesson

The paper concludes that the strength of an "AI Crowd" doesn't come from asking the same smart person to guess a million times. It comes from asking different smart people who make different kinds of mistakes.

The Mistake: Asking the same model to guess 5 times. (Like asking one person to write 5 different stories; they will all sound the same).
The Solution: Asking 5 different models to guess once. (Like asking a poet, a scientist, a comedian, and a historian to tell a story; their different perspectives create a much richer and more accurate picture).

In short: To get the best predictions, don't just stack the "best" model on top of itself. Mix in models that think differently, even if they aren't the absolute #1 on the leaderboard. Diversity is the secret sauce.

Technical Summary: Diversity is the Strength of the AI Crowd

Problem Statement

Current top-performing AI forecasting systems achieve accuracy approaching that of skilled human "superforecasters" on tournament-style questions regarding future world events. These systems typically rely on a standard recipe: wrapping an off-the-shelf Large Language Model (LLM) in a forecasting-specific pipeline (involving retrieval, structured prompting, and multi-sample aggregation).

However, practitioners face a constrained optimization problem rarely addressed in literature: given a fixed budget of forecasts per question, which specific models should be sampled and combined to maximize ensemble accuracy? A natural but incorrect assumption is that drawing all samples from the single strongest model yields the best results. This paper investigates whether this assumption holds and seeks to determine the optimal allocation of a fixed sample budget across a pool of diverse frontier LLMs.

Methodology

The study evaluates ensemble performance on the Metaculus AI Benchmark Q2 2025, utilizing a test set of 113 binary event questions resolved between May and July 2025.

Models and Pipeline

The authors evaluated five forecasters, each wrapped in an identical two-stage scaffolding pipeline (retrieval of question-specific evidence followed by structured probability emission):

Four off-the-shelf frontier LLMs: Gemini 3 Pro, GPT-5, Grok 4, and Kimi K2.5.
One fine-tuned model: FT-gpt-oss-120b (trained via reinforcement learning on a disjoint set of forecasting questions).

Each model generated $R_m = 3$ independent samples per question. The ensemble prediction is a weighted average of the per-model averages, followed by a clipping operation to bound probabilities away from 0 and 1.

Evaluation Metrics

Accuracy: Measured using the Metaculus baseline score (a rescaled log score), where 0 represents a uniform 50% prediction and 100 represents perfect foresight.
Diversity: Quantified using the symmetric Jensen–Shannon (JS) divergence between the Bernoulli predictive distributions of pairs of models.
Ensemble Optimization: The authors performed a brute-force enumeration of all valid integer sample allocations for a total budget of $B=5$ (with a per-model cap of 3 samples), identifying the optimal team $a^*_5$ that maximizes the mean baseline score.
Replaceability: Defined as the score gap ( $\Delta_m$ ) between the optimal team and the best possible team that excludes a specific model $m$ .

Key Results

1. Correlation Limits the Value of Redundant Sampling

The study found that individual accuracy alone is insufficient for maximizing ensemble performance. The strongest off-the-shelf frontier LLMs (e.g., GPT-5, Kimi K2.5) produce highly correlated predictions. Consequently, adding multiple samples from the same high-performing model yields diminishing returns compared to adding samples from a less correlated model.

2. Optimal Allocation Requires Diversity

Under a fixed budget of $B=5$ , the optimal ensemble never concentrates all samples on a single model. Instead, the optimal allocation ( $a^*_5$ ) splits the budget across four distinct models:

FT-gpt-oss-120b: 2 samples
Gemini 3 Pro: 1 sample
GPT-5: 1 sample
Grok 4: 1 sample
Kimi K2.5: 0 samples

3. The "Least Replaceable" Model is Not the Most Accurate

A critical finding concerns the contribution of Grok 4.

In solo performance, Grok 4 ranked third among the five models.
However, in the optimal ensemble, Grok 4 was the least replaceable member, contributing a score gap ( $\Delta_{Grok}$ ) of approximately 1.7 baseline points (BP) if removed.
In contrast, the top two solo performers (GPT-5 and Gemini 3 Pro) contributed less than 0.1 BP each to the ensemble's marginal gain, indicating they are largely interchangeable within the optimal team.

The disproportionate value of Grok 4 stems from its low correlation with the other frontier models, not its solo accuracy. Similarly, the fine-tuned model (FT-gpt-oss-120b) received the highest weight in continuous-weight sweeps and was the second least replaceable, driven by its distinct error profile.

Significance and Claims

The paper claims that the strength of an "AI crowd" in forecasting does not arise from indiscriminately sampling more forecasts from the single best model. Instead, it arises from combining forecasts across models with complementary errors.

Theoretical Alignment: These results empirically validate classical ensemble theory (Krogh & Vedelsby, 1994; Wood et al., 2023), which posits that ensemble error decomposes into an accuracy term and a diversity term. In this setting, where solo accuracies are clustered, the diversity term becomes the primary differentiator.
Practical Implication: For practitioners building forecasting systems, the optimal strategy is to explicitly optimize for both model quality and diversity, rather than simply stacking samples from the highest-ranked model.
Implication for Developers: Models that are not top-ranked in solo accuracy may still provide the highest marginal value to an ensemble if they possess distinct pre-training or fine-tuning that decorrelates their errors from the mainstream frontier models.

The authors conclude that future forecasting systems should be designed to explicitly seek out and weight models that offer complementary error structures, rather than relying solely on aggregate performance metrics of individual models.