Wisdom Of The (AI) Crowd: Investigating Artificial Swarm Intelligence In Large Language Models

Technical Summary: Wisdom of the (AI) Crowd: Investigating Artificial Swarm Intelligence in Large Language Models

Problem Statement
Human swarm intelligence (the "wisdom of the crowd") demonstrates that collective judgments often outperform individual estimates, yet this phenomenon faces significant scalability constraints regarding cost, coordination, time, and susceptibility to social biases. While discriminative AI systems have long utilized ensemble learning (e.g., bagging, boosting) to improve accuracy, a critical gap exists in understanding whether generative AI, specifically Large Language Models (LLMs), can approximate swarm intelligence effects. Current literature lacks systematic investigation into whether aggregating outputs from LLMs—either through repeated sampling of a single model or by combining heterogeneous models—can reduce estimation errors on uncertain tasks in a manner comparable to human collectives.

Methodology
The authors conducted a controlled experiment involving 960 manually executed prompts across three proprietary LLMs: GPT-5, Gemini 2.5 Pro, and Claude Sonnet 4.5. The study utilized a within-subject design with eight estimation tasks balanced between uncertainty and verifiable ground truth:

• Task Composition: Five Fermi problems (requiring order-of-magnitude reasoning for quantities like global income or subway reconstruction costs) and three economic estimation problems (based on anonymized historical financial data, such as stock prices and subscriber counts).
• Aggregation Modes:
  • Intra-model (A1–A3): Each model was prompted 30 times independently per task to capture internal stochastic variance.
  • Inter-model (B1): A heterogeneous swarm combining 10 runs from each of the three models (30 total runs) to assess cross-model diversity.
• Metrics: The study measured Mean Absolute Error (MAE) and Mean Absolute Percentage Error (MAPE) for both individual and aggregated (median/mean) estimates. Additionally, the authors analyzed the calibration of self-reported 90% confidence intervals and calculated Spearman's rank correlation between relative interval widths and relative estimation errors to assess metacognitive awareness.

Key Contributions

1. Conceptualization of LLM Swarms: The paper introduces "LLM Swarms" as a novel construct, applying swarm intelligence principles from human collectives to artificial systems. It defines these as ensembles generated by repeatedly sampling single models or combining multiple models of varying architecture and scale.
2. Proof-of-Concept Methodology: The authors establish a rigorous experimental framework for evaluating LLM swarms on uncertain estimation tasks. This approach avoids trivial data retrieval (by using Fermi problems and anonymized economic data) while maintaining verifiable ground truth.
3. Empirical Evidence of Aggregation Benefits: The study provides the first systematic empirical evidence that aggregating LLM outputs reduces estimation errors. It demonstrates that both intra-model sampling and inter-model heterogeneity yield significant improvements in decision quality.

Results

• Error Reduction: Aggregation consistently reduced errors across all models and tasks. MAPE reductions ranged from negligible improvements to significant gains of up to 37.41 percentage points (observed in Task F2 with the heterogeneous B1 mode).
• Task Dependency: The effectiveness of aggregation was highly contingent on task characteristics. Economic forecasting tasks (Eco1–Eco3), which possessed historical statistical anchors, showed more consistent and substantial improvements than Fermi problems (F1–F5), which required abstract reasoning without reference points.
• Model Variance: Claude Sonnet 4.5 exhibited low internal variance (high consistency), which limited the benefits of intra-model aggregation compared to GPT-5 and Gemini 2.5 Pro, which showed greater stochastic diversity.
• Confidence Intervals: While individual models often failed to achieve the expected 90% calibration rate, the heterogeneous aggregation (B1) successfully captured the ground truth within the aggregated confidence interval for all eight tasks.
• Metacognitive Awareness: A positive correlation (Spearman's $\rho$ = 0.242–0.568, all $p < 0.001$) was observed between the width of self-reported confidence intervals and estimation errors. This suggests LLMs possess a form of metacognitive awareness, assigning wider intervals to more uncertain estimates, with GPT-5 showing the strongest correlation ($\rho = 0.568$).

Significance and Claims
The paper claims to establish a new research stream at the intersection of collective intelligence and generative AI. It posits that LLM swarms offer a scalable, low-cost, and rapid alternative to human crowds for decision support, particularly in time-sensitive contexts. The authors argue that while LLMs may not perfectly replicate human swarm dynamics due to shared training biases and "echo chamber" effects in single-model sampling, heterogeneous aggregation can effectively mitigate individual model errors.

The study modestly frames its findings as initial evidence rather than a definitive solution. It highlights that the "wisdom of the AI crowd" is not universal but depends on task structure (e.g., presence of statistical anchors) and model diversity. The authors conclude that understanding these aggregation strategies is critical for organizations deploying genAI, suggesting that artificial swarms can provide "second-order swarm intelligence"—aggregating systems that are themselves products of collective human knowledge. However, they caution that limitations regarding hallucinations, black-box reasoning, and the lack of true cognitive diversity (compared to humans) remain significant challenges for high-stakes applications.