Human Capital, Not Model Benchmarks, Predicts Hybrid Intelligence in Forecasting

Technical Summary: Human Capital, Not Model Benchmarks, Predicts Hybrid Intelligence in Forecasting

Problem Statement
Current literature on human–AI collaboration yields conflicting results regarding productivity and accuracy. Some studies suggest AI augments lower-skilled workers, while others indicate benefits accrue only to experienced users, or that access to capable models actually degrades human performance relative to the model working alone. A recurring confound in these studies is the evaluation metric: "ability" is often measured against problems where the AI has already encoded the solution (contamination), or "quality" is scored by raters susceptible to the fluency of AI-generated prose rather than substantive reasoning. This pilot study addresses the need for an objective, externally resolved benchmark to determine whether human capital moderates the value of AI assistance in forecasting and whether distinct interaction styles mediate this effect.

Methodology
The study utilized a real-money prediction market (Polymarket) as an objective ground truth, avoiding the biases of expert raters or static benchmarks.

• Participants and Design: 78 participants (42 UC Berkeley students, 36 community adults) were organized into 26 three-person teams. They were assigned to either a Human-only condition (12 teams) or a Hybrid condition (14 teams) with access to one of four Large Language Models (Llama 3.1, Qwen3, GPT-4o, Gemini 3 Pro). A separate pool of 30 participants worked with a "Socratic" model that withheld answers.
• Task: Teams forecasted 10 randomly drawn questions from a pool of 30 live Polymarket contracts (resolving Nov 2025–Jan 2026) covering economics, international relations, and business.
• Metrics: Forecast accuracy was measured using the scaled Brier score (mean squared error of probabilistic forecasts; lower is better).
• Human Capital Measures: Participants completed assessments for fluid reasoning (Raven's Matrices), general intelligence ($g$), perspective-taking (IRI), intellectual humility, epistemic curiosity, resilience, and Big-Five personality traits.
• Interaction Styles: Post-hoc classification of hybrid participants into three emergent styles:
  • Automators: Adopt the model's answer with minimal reasoning.
  • Validators: Use the model to check a prior guess.
  • Cyborgs: Engage in iterative, complementary reasoning.
• Analysis: Analyses were conducted at the individual forecaster level using non-parametric omnibus tests (Kruskal–Wallis) and Welch's t-tests for pairwise comparisons, alongside Pearson correlations for human capital traits.

Key Results
The study identified a "trimodal" distribution of hybrid performance, challenging the notion of a single average effect for human–AI collaboration:

1. Human-Only: Scored 14.7 (Brier), performing significantly worse than the models but better than chance.
2. Automators: Scored 10.4. They reproduced the model's behavior but failed to match its accuracy, performing worse than the AI-only baseline (5.5).
3. Validators: Scored 31.7. This group, using AI to rubber-stamp prior guesses, performed significantly worse than humans working alone and far below the AI baseline.
4. Cyborgs: Scored 3.8. This minority group achieved error rates lower than every individual AI model and were statistically indistinguishable from, and numerically superior to, the market benchmark (3.5).

Predictors of Success

• Decoupling of Ability and Outcome: In the Human-only condition, raw cognitive ability ($g$, fluid reasoning) strongly predicted accuracy ($r \approx -0.45$). However, in the Hybrid condition, raw cognitive ability did not predict accuracy ($r \approx -0.10$). The model supplied a performance floor, decoupling raw intelligence from the final outcome.
• Role of Collaborative Human Capital: Success in the Hybrid condition was predicted by collaborative traits rather than benchmarks. Perspective-taking ($r = -0.32$), curiosity, and intellectual humility were the primary differentiators.
• Cyborg Emergence: The "Cyborg" mode was sharply gated by human capital. Cyborgs exhibited significantly higher levels of intellectual humility, curiosity, and perspective-taking compared to other hybrid forecasters (effect sizes $d > 3.0$).

Significance and Claims
The paper posits that the question of AI productivity must shift from "whether AI helps" to "when and how it helps." The primary contribution is the demonstration that pooling all hybrid interactions into a single average obscures a strongly trimodal outcome, including a "Validator" mode that actively degrades performance.

The study claims that collaborative human capital (specifically perspective-taking, intellectual humility, and curiosity) is the critical moderator that determines whether a human–AI team succeeds or fails. Crucially, the traits that make a model perform well or a human reason well in isolation are distinct from the traits that make a human–AI team effective. The results are presented as preliminary but statistically robust, serving to motivate a pre-registered replication currently in preparation to confirm these effect sizes and causal mechanisms.

Limitations
The authors explicitly note the pilot nature of the work: small sample sizes for the Cyborg and Validator groups ($n=9$ each), the emergent (non-manipulated) nature of interaction styles which creates collinearity with human capital, and low within-condition variance. Consequently, comparisons across styles are descriptive rather than causal, and the study aims to generate effect-size estimates for a future confirmatory trial.