🤖 AI

Wisdom of LLM Crowds: Aggregation and Contamination in Language Model Ensembles

This paper demonstrates that aggregating predictions from diverse large language models via learned methods can surpass individual model performance, while highlighting that training cutoff contamination significantly distorts capability assessments and that the primary benefit of such ensembles stems from linearly combining diverse model outputs rather than complex nonlinear interactions.

Original authors: Igor Douven

Published 2026-07-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Igor Douven

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where the smartest person in the room isn't actually the smartest at all. Instead, the real genius is the whole group sitting around the table. This idea, known as the "wisdom of crowds," suggests that if you take the guesses of many different people and average them out, the result is often more accurate than the guess of the single best expert. Think of it like a flock of birds: no single bird sees the whole picture, but together they navigate perfectly. Scientists have known for a long time that this works for humans. But now, with Artificial Intelligence (AI) everywhere, a big question has popped up: Does this magic trick work for robots, too? If you ask a bunch of different AI models to guess the outcome of a future event, will their combined answer be smarter than any single AI? This isn't just a fun game; it matters because if AI crowds work, we could use them to predict everything from weather to stock markets without needing a team of human experts. But there's a catch: AI models are trained on data from the past, and if they've already "seen" the answer in their training data, they aren't really guessing—they're just cheating by remembering.

This paper dives into that exact question, treating 15 different Large Language Models (LLMs) like a team of contestants in a prediction game. The researchers asked these AIs to guess the outcomes of 254 real-world questions, similar to betting on whether a sports team will win or a politician will get elected. They then tried to combine the AIs' answers in different ways to see if the group could beat the individuals. The results were a mix of exciting news and a major warning label.

First, the good news: The "wisdom of crowds" does seem to work for AI, but only if you combine them correctly. The researchers found that simply taking the average of all the AI guesses wasn't the best strategy. Instead, they used a clever computer program (a "learned aggregator") to figure out how to mix the answers. This smart mixer did a better job than any single AI model, and even better than the simple average. Interestingly, the study suggests that this improvement didn't come from the AI mixer doing some complex, magical math. Instead, it worked because it learned to give more weight to the models that made different kinds of mistakes. It's like a sports coach realizing that if one player is great at defense but bad at offense, and another is the opposite, putting them together covers all the bases. The study found that a simple linear combination—basically a weighted average—was enough to get the best results, suggesting that the key to AI crowd wisdom is diversity in errors, not complex interactions.

However, there is a massive "but" that changes everything. The researchers discovered that many of the AI models were secretly cheating. Because these models are trained on huge chunks of the internet up to a certain date, they often "remembered" the answers to questions that were resolved before their training stopped. This is called "contamination." When the researchers filtered out all the questions the AIs might have already known the answers to, the results changed dramatically. The huge gap between the fancy, expensive "cloud" models and the smaller, local models almost vanished. The big models looked much less impressive when they couldn't rely on their memory. In fact, even the best AI crowd was still significantly worse than a real human prediction market, where people bet money on outcomes in real-time. The human market was about twice as accurate as the AI crowd, even when the humans were looking at the same information the AIs had.

So, what's the takeaway? AI crowds can be smart, but they are easily fooled by their own memories. If you want to know if an AI is actually good at predicting the future, you have to test it on things that happened after it finished learning. Until then, the "wisdom" of an AI crowd might just be a reflection of what it already knows, rather than what it can figure out. The study suggests that while we can build better AI teams by mixing different models, we still have a long way to go before they can match the dynamic, real-time intelligence of a human crowd.

Technical Summary: Wisdom of LLM Crowds

Problem Statement

The "wisdom of crowds" phenomenon posits that aggregating judgments from a diverse group often yields higher accuracy than the best individual member. While extensively validated with human forecasters, it remains an open question whether Large Language Models (LLMs) exhibit similar collective intelligence. Previous work (e.g., Schoenegger et al., 2024) suggested LLM crowds could match human panels, but relied on simple aggregation (median) and did not systematically investigate the mechanisms of aggregation benefits or the impact of data contamination.

A critical challenge in evaluating LLMs on resolved prediction market questions is training cutoff contamination. If a model's training data includes the outcomes of questions resolved before its cutoff date, the model may "memorize" the answer rather than reason, artificially inflating performance and distorting comparisons between models with different training dates. This study aims to determine if LLM crowds exhibit wisdom-of-crowds effects, identify the optimal aggregation strategies, and rigorously assess the impact of contamination on model rankings and capability gaps.

Methodology

Data and Models

Dataset: The study utilized 254 binary (YES/NO) prediction market questions from Manifold Markets.
- Main Set: 208 items resolving between June 2024 and March 2026.
- Clean Subset: 94 items resolving after September 1, 2025, ensuring no overlap with any model's training cutoff.
Models: Probability estimates were elicited from 15 LLMs, comprising:
- Frontier Cloud Models: 10 models (e.g., Claude 4.6 series, GPT-5 series, Gemini Flash/Pro).
- Local Models: 5 smaller, open-weight models (e.g., Mistral 7B, LLaMA 3.1 8B, Phi-4 14B).
Elicitation: Models were prompted to provide a probability estimate (0–1) for each statement based solely on training knowledge, without internet search.

Aggregation Methods

The study evaluated four classical aggregation methods and two learned aggregators:

Classical Methods: Arithmetic mean, Geometric mean, Harmonic mean, and Log-odds mean (logit mean).
Learned Aggregators:
- Logistic Regression: A linear model learning non-uniform weights for the 15 model inputs.
- Multilayer Perceptron (MLP): A neural network with two hidden layers (64 and 32 units) trained to map the 15-dimensional input vector to a single probability.

Evaluation Protocol: Learned aggregators were trained on the 160 non-clean items and evaluated on the 94-item clean subset (temporal transfer) to ensure they were tested on unseen data. Classical methods were evaluated directly on the clean subset.

Analysis Techniques

Metrics: Primary metric was the Brier score (lower is better); secondary metrics included classification accuracy and AUC.
Symbolic Regression: Applied to the trained MLP to recover an interpretable mathematical formula approximating its aggregation strategy, searching for the Pareto frontier of accuracy vs. complexity.
Contamination Analysis: Compared model performance on "within-cutoff" vs. "outside-cutoff" items and analyzed rank stability between the full dataset and the clean subset.
Human Baseline: Compared LLM performance against the Manifold market probability, using both the final market price and a "cutoff-matched" price (market price at the time of each model's training cutoff) to control for information advantages.

Key Results

1. Aggregation Performance

Learned vs. Classical: Both learned aggregators (Logistic Regression and MLP) significantly outperformed all individual models and classical aggregation methods on the clean subset.
- Logistic Regression achieved the lowest Brier score (0.241), outperforming the arithmetic mean (0.313) by ~23%.
- MLP achieved a slightly higher Brier score (0.264) but better rank-order discrimination (AUC 0.657 vs. 0.633).
Linearity of Benefit: The near-equivalence of the Logistic Regression and MLP suggests the aggregation benefit derives primarily from learning a linear combination of diverse model outputs rather than complex nonlinear interactions.
Symbolic Regression Findings: The simplest useful formula recovered from the MLP was a contrastive signal between two specific models: $\sigma(p_{\text{Gemini Pro}} - p_{\text{Claude Haiku}})$ . This formula, using only two models, nearly replicated the performance of the full 15-model MLP, indicating that the benefit stems from a small number of complementary error patterns.

2. Contamination Effects

Performance Inflation: Models showed significantly higher accuracy on items resolving before their training cutoff compared to those resolving after. Accuracy differentials ranged from +0.05 to +0.26.
Rank Instability: Contamination substantially distorted model rankings. The Spearman rank correlation between the full dataset and the clean subset was only $\rho = 0.532$ . Notable shifts included LLaMA 3.1 8B rising from rank 14 to 6, and GPT-5.4 Mini dropping from 6 to 13.
Cloud-Local Gap Collapse: The apparent performance gap between frontier cloud models and smaller local models collapsed from 35.8% (on the full dataset) to 8.9% (on the clean subset). This suggests much of the perceived superiority of frontier models in forecasting tasks was due to knowledge of outcomes rather than superior reasoning.

3. Comparison with Human Crowds

LLM vs. Market: Even when controlling for the market's information advantage (using cutoff-matched market prices), the human crowd (Manifold market) outperformed every individual LLM by a factor of 1.5x to 2.6x in Brier score.
Mechanism: The human advantage reflects the ability of active traders to dynamically aggregate dispersed knowledge and update estimates in real-time, a capability LLMs (reasoning from frozen snapshots) cannot replicate.
Reconciliation with Prior Work: The study notes that while LLM crowds fall short of dynamic prediction markets, they can match the performance of static human panels (as found by Schoenegger et al., 2024), highlighting that the gap is structural (dynamic vs. static aggregation) rather than a fundamental failure of LLMs relative to human cognition.

Significance and Claims

The paper claims three primary contributions:

Validation of LLM Crowd Wisdom: It provides evidence that LLM crowds exhibit wisdom-of-crowds effects, where learned aggregation significantly outperforms the best individual model.
Mechanism Identification: It demonstrates that the aggregation benefit in LLMs, similar to human crowds, relies on error-pattern diversity rather than simple performance weighting. The authors show that simple linear combinations of diverse models are sufficient to capture most of this benefit.
Contamination as a Critical Confound: The study establishes that training cutoff contamination is a pervasive issue that artificially inflates the performance of frontier models and distorts capability assessments. It argues that "clean" evaluation (using questions resolving after all models' training cutoffs) is essential for reliable assessment, revealing that the gap between local and frontier models is far smaller than previously thought.

The authors conclude that while LLM crowds can exhibit collective intelligence, their accuracy is structurally limited by their static nature compared to dynamic human markets, and that rigorous, contamination-free evaluation is necessary to understand their true reasoning capabilities.