Preference Optimization Drives Monoculture in LLM Prediction Markets

Technical Summary: Preference Optimization Drives Monoculture in LLM Prediction Markets

1. Problem Statement

Prediction markets rely on the theoretical assumption that participant errors are independent; under this condition, market prices converge to the crowd's best estimate (Wolfers & Zitzewitz, 2004; Hanson, 2003). However, as Large Language Model (LLM) agents become active traders on platforms like Kalshi and Polymarket, a critical question arises: does error independence hold when the "crowd" consists of LLMs sharing the same training pipeline?

Existing literature attributes correlated errors to behavioral phenomena like herding or information cascades. This paper posits a structural cause: alignment training itself, specifically Direct Preference Optimization (DPO), forces models toward a shared preferred-output distribution. This creates a "preference-optimization monoculture" where agents, despite being distinct instances, fail and succeed together, undermining the collective intelligence of the market.

2. Methodology

The authors evaluate this hypothesis using a controlled experimental setup:

• Market Mechanism: A Logarithmic Market Scoring Rule (LMSR) with a liquidity parameter $b=100$ and two outcomes.
• Trading Protocol: 50 binary questions (TruthfulQA) are posed to $N$ agents (ranging from 1 to 40). Agents trade sequentially, betting only when their stated confidence exceeds the current market price. Wealth persists across questions, allowing accurate agents to accumulate capital.
• Metrics:
  • Pairwise Error Correlation ($\rho$): Pearson correlation between agents' binary error vectors.
  • Effective Number of Forecasters ($N_{eff}$): Calculated as $N_{eff} = \frac{N}{1 + (N-1)\rho}$. This metric quantifies the independent forecasting power of a group of $N$ correlated agents.
• Experimental Variables:
  • Scaling: Testing $N \in \{5, 10, 20, 40\}$ agents of the same model.
  • Model Diversity: Comparing same-model pairs vs. cross-model pairs (Llama, Qwen, Mistral, GLM).
  • Causal Ablations: Isolating the impact of specific training stages (Pre-training/ICL, Supervised Fine-Tuning (SFT), and DPO) using identical weight checkpoints from the Princeton NLP and AllenAI Tulu 3 pipelines.
  • Mitigation Strategies: Testing temperature diversity, role diversity (epistemic roles), and cross-model mixing.

3. Key Contributions and Results

A. The Monoculture Effect
The study finds that LLM agents fine-tuned with DPO exhibit high pairwise error correlations ($\rho \approx 0.70$).

• Diminishing Returns: Ten same-model agents provide the forecasting power of only $\approx 1.4$ independent forecasters ($N_{eff} \approx 1.38$).
• Flat Accuracy: Increasing the number of agents from 5 to 40 does not improve market accuracy (staying flat at 66.0–69.6%), which fails to match a single standalone agent (70.2%).
• Error Distribution: Empirical data shows a spike in unanimity (all agents right or all wrong) compared to the binomial distribution expected under independence.

B. Causal Identification of DPO
Through controlled ablations, the paper isolates Preference Optimization (DPO) as the primary driver of this correlation:

• Princeton NLP Ablation: Using identical SFT weights, the transition to DPO increased correlation by $\Delta\rho = +0.46$ (though the SFT baseline had near-chance accuracy).
• Tulu 3 Ablation: At meaningful accuracy levels, the transition from SFT to DPO increased correlation by $\Delta\rho = +0.24$ (8B model) and $+0.28$ (70B model).
• Conclusion: Pre-training creates a "knowledge monoculture," but the alignment pipeline (specifically DPO) amplifies this into an "answer monoculture" by pushing policies toward the mode of the reward model's preferred distribution.

C. Mitigation Strategies
The paper tests three strategies to reduce correlation:

1. Cross-Model Diversity: Mixing different model families (e.g., Llama and Qwen2.5) reduces $\rho$ from 0.68 to 0.40, raising $N_{eff}$ from 1.4 to 2.2. This is the most effective mitigation.
2. Role Diversity: Assigning distinct epistemic roles (skeptic, Bayesian, etc.) reduces $\rho$ to 0.44 without accuracy loss.
3. Temperature Diversity: Increasing temperature reduces $\rho$ but incurs a significant accuracy penalty (4.4% drop).

D. Adversarial Robustness

• LMSR vs. Debate: In adversarial settings (e.g., 1 honest vs. 9 malicious agents), LMSR markets outperform multi-agent debate. Debate collapses under adversarial pressure (44.0% accuracy at 1v9), whereas LMSR maintains higher accuracy (59.6%) due to the convex cost function of LMSR, which imposes prohibitive capital costs on overturning a strong consensus.
• Self-Deterrence: Adversaries equipped with a price-threshold skip rule voluntarily decline trades when the honest consensus is strong, further stabilizing the market.

4. Significance and Claims

The paper claims that the integrity of LLM-based prediction markets is fundamentally compromised by the standard alignment pipeline.

• Structural Failure: The failure of the "wisdom of crowds" in LLM markets is not a scaling problem but a structural consequence of preference optimization.
• Integrity Metric: The authors propose $N_{eff}$ as a critical integrity metric for market platforms. A market of 1,000 same-model agents provides effectively the same information as $\approx 1.4$ independent agents.
• Diversity Requirement: As LLMs become more aligned, markets built from them become more monocultural. To restore market efficacy, platforms must prioritize cross-model diversity or role diversity rather than simply scaling the number of identical agents.

The paper concludes that while LMSR provides robustness against adversarial manipulation, it cannot overcome the inherent correlation introduced by shared alignment training, necessitating a shift in how multi-agent systems are constructed for forecasting tasks.