Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefs

1. Problem Definition

The paper addresses the challenge of binary probabilistic forecasting (estimating the probability $P(Y=1)$ of a future event) using Large Language Models (LLMs). The authors focus on two main categories of questions:

• Judgemental Forecasting: Open-ended text questions (e.g., "Will Person X be President?") requiring synthesis of news and geopolitical context.
• Time-Series Forecasting: Numerical questions (e.g., "Will Stock Y be higher than value V?") requiring analysis of historical data trends.

The evaluation is conducted on ForecastBench, a standardized benchmark featuring questions from market sources (Polymarket, Manifold) and dataset sources (FRED, yfinance, Wikipedia). The primary goal is to achieve State-of-the-Art (SOTA) performance while rigorously preventing information leakage (the model accessing data post-cutoff date) during backtesting.

2. Methodology: Bayesian Linguistic Forecaster (BLF)

The core contribution is BLF, an agentic system built on three pillars:

A. Linguistic Belief State (Sequential Bayesian Updating)

Unlike standard approaches that append raw search results to an ever-growing context window (text accumulation) or perform batch search followed by reasoning, BLF maintains a semi-structured belief state updated at every step of an iterative tool-use loop.

• Structure: At each step $t$, the LLM generates an action $a_t$ (e.g., search, read file) and an updated belief $b_t$.
• Belief Content: $b_t$ is a JSON object containing:
  • A probability estimate $p \in [0, 1]$.
  • A confidence level.
  • Natural language summaries of evidence for and against the outcome.
  • Open questions for future investigation.
• Mechanism: This acts as approximate sequential Bayesian inference. The LLM integrates new evidence ($o_t$) into the belief state ($b_t$) before deciding the next action, allowing for coherent, iterative refinement rather than one-shot reasoning.

B. Hierarchical Multi-Trial Aggregation

To mitigate the high variance inherent in LLM generations:

• K-Trials: The system runs $K=5$ independent trials per question.
• Logit-Space Averaging: Forecasts are aggregated by averaging in logit space: $\hat{p} = \sigma(\frac{1}{K}\sum \text{logit}(p_k))$. This preserves extremity better than arithmetic mean.
• Shrinkage: The system explores James-Stein-style shrinkage toward $p=0.5$ (controlled by a parameter $\alpha$) to reduce overconfidence when trials disagree. On ForecastBench, optimal $\alpha=1$ (no shrinkage) was found, but shrinkage proved beneficial on smaller, more diverse datasets.

C. Hierarchical Calibration

To ensure forecasts are well-calibrated (i.e., a 70% prediction happens 70% of the time):

• Platt Scaling: A post-processing step maps raw probabilities to calibrated ones.
• Hierarchical Approach: Instead of a global Platt scaling (which can over-shrink extreme predictions from sources with skewed base rates), BLF uses hierarchical Platt scaling with per-source intercept offsets ($\delta_s$). This allows the system to learn specific calibration biases for different data sources (e.g., Polymarket vs. Wikipedia).

D. Robust Backtesting Framework

A critical component is the four-layer leakage defense to ensure validity:

1. Search Engine Filtering: Restricting search results to dates before the forecast cutoff.
2. LLM Leak Classifier: A secondary LLM scans search results for content that implies post-cutoff knowledge.
3. Date Clamping: Tool APIs (e.g., time-series fetchers) are algorithmically clamped to the cutoff date.
4. URL Blocking: Blocking access to resolution sources (e.g., specific market URLs) that might reveal the answer.

• Result: A post-hoc audit showed a residual leakage rate of only 1.5%.

3. Key Contributions

1. SOTA Performance: BLF achieves the highest Brier Index (BI) on ForecastBench, outperforming top public methods (Cassi, GPT-5, Grok 4.20, Foresight-32B) and human superforecasters.
2. Structured Belief State: Demonstrated that maintaining a structured, semi-structured belief state is nearly as impactful as web search access itself. Removing it degraded performance by $\Delta BI = -3.0$, comparable to removing search ($\Delta BI = -4.6$).
3. Rigorous Statistical Methodology: The paper introduces a paired analysis framework with bootstrap confidence intervals to control for question difficulty (which accounts for ~62% of variance), ensuring fair comparisons between methods.
4. Leakage Defense: Established a rigorous standard for backtesting with a verified low leakage rate, addressing a major flaw in previous LLM forecasting evaluations.

4. Results

• Overall Performance: On 400 questions (791 resolution dates) from ForecastBench, BLF achieved a Brier Index (BI) of 73.8, significantly outperforming the next best method (Cassi, BI=70.8).
• Market Questions: BLF is the only method that significantly beats the "crowd" baseline (market price). All other methods, including those with search tools, were statistically indistinguishable from simply returning the market price.
• Dataset Questions: BLF significantly outperformed the empirical prior baseline and other LLMs.
• Ablation Studies:
  • Search: Removing web search reduced BI by 4.6.
  • Belief State: Removing the structured belief state reduced BI by 3.0.
  • Sequential vs. Batch: Replacing the sequential loop with a batch-search-then-reason approach reduced BI by 3.8.
  • Calibration: Hierarchical calibration provided significant gains for zero-shot baselines but had minimal effect on the full BLF system (which was already well-calibrated).
• Human Comparison: BLF's difficulty-adjusted Brier Index (ABI = 71.0) matched the median performance of human superforecasters (ABI = 70.9).

5. Significance

This paper represents a significant leap in agentic AI for forecasting. It moves beyond simple "prompt engineering" or "retrieval-augmented generation" (RAG) by introducing a cognitive architecture where the agent maintains a dynamic, structured internal state (belief) that evolves through iterative tool use.

The findings suggest that:

1. Iterative reasoning with structured belief tracking is more effective than accumulating raw context.
2. Ensembling multiple trials and hierarchical calibration are essential for robust probabilistic outputs.
3. Rigorous backtesting is critical; without strict leakage controls, LLM performance metrics are inflated and unreliable.

BLF sets a new benchmark for how LLMs can be utilized as "superforecasters," combining the reasoning capabilities of agents with the statistical rigor of Bayesian updating, offering a blueprint for future applications in finance, geopolitics, and risk assessment.