Technical Summary: Diverse Evidence, Better Forecasts: Multi-Agent Deliberation Under Information Asymmetry

Problem Statement

Multi-agent systems utilizing Large Language Models (LLMs) are increasingly employed for forecasting future events, operating under the assumption that deliberation among multiple agents improves reasoning and calibration. However, existing approaches typically provide all agents with identical evidence corpora. The authors identify a critical flaw in this design: when agents share identical information, deliberation collapses into "herding" rather than genuine belief revision. In such homogeneous settings, agents reach the same conclusions independently, and iterative exchange merely reinforces shared priors. Recent theoretical work suggests that under identical inputs, multi-agent debate behaves as a martingale where expected accuracy does not improve over rounds, and empirical studies show that LLM errors are highly correlated (often >60%) across different providers. Consequently, multi-agent systems with homogeneous input offer little advantage over single-agent baselines.

Methodology: InfoDelphi and Designed Information Asymmetry

To address the correlation of errors in homogeneous settings, the authors propose designed information asymmetry, treating information asymmetry not as a limitation but as a design principle to exploit. They instantiate this principle in InfoDelphi, a multi-agent forecasting framework.

1. Theoretical Foundation

The authors formalize the benefit of information asymmetry using the bias-variance-covariance decomposition of ensemble mean squared error (MSE).

• Correlated Errors: When agents receive identical evidence ($E$), their prediction errors are driven by the same sources, leading to high inter-agent error correlation ($Cov(\epsilon_i, \epsilon_j) \approx Var(\epsilon)$). This causes the diversity term in the ensemble MSE to collapse, negating the benefits of aggregation.
• Decorrelation via Partitioning: The authors propose partitioning the evidence corpus $E$ into a shared public subset ($E_{pub}$) and disjoint private subsets ($E_{priv}$).
  • Let $\rho$ be the ratio of public evidence.
  • Theoretical analysis (Proposition 3.1) demonstrates that inter-agent error correlation decreases monotonically as $\rho$ decreases from 1 (all public) to 0 (all private).
  • Specifically, with a public ratio of $\rho=0.5$ and equal variance components, inter-agent correlation drops to 0.5 (half the baseline of 1.0).
• Rationale Sharing: To ensure agents can benefit from each other's private signals, the framework requires the exchange of rationales (reasoning traces) rather than just numeric probability estimates. The authors argue via the Data Processing Inequality that exchanging only numbers forms a Markov chain where information about the ground truth can only decrease. Sharing rationales breaks this chain, allowing private evidence to propagate across agent boundaries.

2. The InfoDelphi Framework

InfoDelphi operates in three stages:

1. Evidence Routing: The evidence corpus is partitioned using BM25 relevance ranking.
   • The top-scoring documents (most relevant to the query) form the shared public pool ($E_{pub}$), providing common ground for communication.
   • The remaining documents are assigned to agents in a round-robin fashion based on relevance scores, creating disjoint private subsets ($E_{priv}$). This ensures each agent holds unique peripheral evidence while sharing core context.
2. Iterative Deliberation: Agents engage in multiple rounds of forecasting. In each round, an agent conditions on its private view ($E_{pub} \cup E_{priv}$) and a history summary of other agents' previous forecasts, which includes structured rationales.
3. Confidence-Weighted Aggregation: Final forecasts are aggregated in logit space. The aggregation weights ($w_j$) are determined by the extremity of the agent's prediction ($|logit(p_j)|$). This mechanism amplifies the predictions of agents who hold stronger private evidence, as they tend to produce more extreme (confident) forecasts.

Key Contributions

1. Designed Information Asymmetry: The paper proposes treating information asymmetry as a first-class design principle for multi-agent reasoning, theoretically proving that public/private evidence partitioning reduces inter-agent error correlation.
2. InfoDelphi Framework: The authors introduce a concrete framework implementing this principle through relevance-aware evidence routing, rationale-based iterative deliberation, and confidence-weighted aggregation.
3. POLYGYM Benchmark: To isolate information utilization from retrieval quality, the authors construct POLYGYM, a controlled benchmark of 375 binary forecasting questions derived from real-world prediction markets (Polymarket). Crucially, all methods operate over the same fixed, pre-retrieved evidence pool, eliminating variability from search strategies.

Experimental Results

Experiments were conducted on POLYGYM using $J=3$ agents and $R=2$ deliberation rounds.

• Performance: InfoDelphi achieved a Brier score of 0.178 and 77.9% accuracy. This outperforms the strongest single-agent baselines (e.g., Chain-of-Thought, Superforecaster) and multi-agent baselines (e.g., Standard Debate, MoA) by 12–18% in Brier score and 4–8 percentage points in accuracy.
• Necessity of Diversity: Ablation studies confirmed that removing information asymmetry (setting $\rho=1.0$, i.e., all public) eliminates most deliberation gains, reducing performance to levels comparable to standard debate.
• Importance of Rationale Sharing: Removing rationale sharing (exchanging only numbers) degraded performance significantly (Brier score increased to 0.211), validating the theoretical claim that closed-system numeric exchange cannot improve collective accuracy.
• Generalization: The framework demonstrated consistent improvements across different model backends, including GPT-5.4-mini, DeepSeek-V3.2, and Llama-4-Scout, with gains scaling with model capability.
• Mechanism Validation: Analysis showed that confidence-weighted aggregation works because agents with stronger private evidence produce more extreme predictions. Furthermore, information asymmetry preserved inter-agent diversity (only 73% variance reduction) compared to homogeneous settings (90–92% variance reduction), preventing the system from collapsing into a consensus echo chamber.

Significance and Claims

The paper claims that diversity of input is the key enabler of effective multi-agent reasoning. The authors argue that the standard practice of providing identical context windows to all agents fundamentally limits the value of multi-agent interaction. By introducing designed information asymmetry, InfoDelphi converts redundant deliberation into productive belief revision.

The authors position probabilistic forecasting as a demanding test of this principle, requiring the synthesis of noisy, contradictory evidence. They suggest that the robust improvements observed in this setting indicate broad applicability to other multi-agent systems where agents reason over shared evidence, such as collaborative question answering and scientific hypothesis evaluation.

The paper maintains modesty regarding its limitations, acknowledging that the theoretical proof relies on idealizing assumptions (linear error decomposition, independence of private noise) and that the current implementation uses standard off-the-shelf methods for routing and aggregation, leaving room for more sophisticated alternatives. Additionally, the authors note that the framework's upper bound is still constrained by the quality of the initial evidence retrieval.