Predictive Coherence and the Moment Hierarchy: Martingale Posteriors for Exchangeable Bernoulli Sequences

This paper demonstrates that for exchangeable Bernoulli sequences, the first posterior moment alone is insufficient to uniquely determine multi-step predictive probabilities, revealing that martingale posterior frameworks generally fail to identify these distributions unless the conditional law of the terminal value is fully specified, as exemplified by Hill's A_n rule under a Jeffreys prior.

The Gist

Here is an explanation of the paper "Predictive Coherence and the Moment Hierarchy," translated into everyday language using analogies.

The Big Picture: The "Guessing Game" of the Future

Imagine you are a weather forecaster trying to predict if it will rain tomorrow, the day after, and the day after that. You have a secret "truth" about the weather (let's call it $\theta$), but you don't know it exactly. You only have a hunch based on past data.

In the world of statistics, there are two main ways to handle this uncertainty:

1. The Full Bayesian Way: You have a complete map of every possible version of the truth. You know not just the average chance of rain, but exactly how "spread out" your uncertainty is.
2. The Martingale Way (The Paper's Focus): You only promise to keep your "average guess" consistent. If you guess 40% rain today, and it rains tomorrow, your new average guess should be higher, but your current average must be a fair prediction of your future average. You don't necessarily have a full map; you just have a rule for updating your average.

The Paper's Main Discovery:
The authors, Polson and Zantedeschi, found a hidden trap in the "Martingale Way."

• If you only care about tomorrow (1-step prediction), knowing your current average guess is enough.
• But if you want to predict tomorrow AND the day after (2-step prediction), knowing just the average is not enough. You are missing crucial information about the "shape" of your uncertainty.

---

Analogy 1: The Two Dice (Why the Average Isn't Enough)

Imagine you are betting on the roll of a die, but you don't know which die is being used. You only know the average roll is 3.5.

• Scenario A (The "Safe" Die): The die is a standard, fair die (1, 2, 3, 4, 5, 6). The average is 3.5.
• Scenario B (The "Wild" Die): The die only has 1s and 6s. It rolls a 1 half the time and a 6 half the time. The average is also 3.5.

The 1-Step Problem:
If I ask, "What is the chance the next roll is a 6?"

• In Scenario A, it's 1/6.
• In Scenario B, it's 1/2.
• Wait, the paper says the average is enough for 1-step? Actually, for a Bernoulli (Yes/No) sequence, the "average" is the probability of the next event. So for the very next step, the average tells you everything you need to know about the next single event.

The 2-Step Problem (The Trap):
Now, I ask: "What is the chance of rolling two 6s in a row?"

• Scenario A (Fair Die): The rolls are independent. Chance = $(1/6) \times (1/6) = 1/36$.
• Scenario B (Wild Die): If you roll a 6, the next roll is also likely to be a 6 (because the die is either all 1s or all 6s). The chance is much higher!

The Lesson:
Even though both scenarios have the exact same average (3.5), they have completely different variances (how much they jump around).

• The "Fair Die" has low variance (it stays close to the average).
• The "Wild Die" has high variance (it swings wildly).

The paper proves that if you only track the average (the Martingale condition), you cannot tell the difference between the "Fair Die" and the "Wild Die." Therefore, you cannot accurately predict two steps ahead. You need to know the variance (the "spread" or "curvature" of your belief).

Analogy 2: The Foggy Mountain (The "Shape" of Belief)

Imagine you are standing on a mountain peak in thick fog. You are trying to guess where the summit is.

• The Average (Mean): You point your finger and say, "The summit is roughly 100 meters away."
• The Full Belief (Posterior): You also know the shape of the fog.
  • Case 1: The fog is a tight, thin cloud. You are very sure the summit is exactly 100m away.
  • Case 2: The fog is a giant, swirling cloud. The summit could be 50m away, or 150m away. The average is still 100m, but you are very unsure.

The Prediction:
If you need to walk one step, both clouds look the same. You take a step toward 100m.
But if you need to walk three steps in a straight line without turning:

• In Case 1 (Tight fog), you can walk confidently.
• In Case 2 (Swirling fog), you might walk 150m and hit a cliff, or 50m and fall into a ravine.

The paper argues that the "Martingale" method only gives you the direction (the average). It doesn't tell you if the fog is tight or swirling. Without knowing the "shape" of the fog (the higher moments), your prediction for a long walk (multi-step prediction) is flawed.

The "Plug-in" Mistake

The paper also critiques a common shortcut statisticians use called the "Plug-in" rule.

• The Rule: "Just take my current best guess (the average) and pretend that's the absolute truth for the future."
• The Result: This is like assuming the "Wild Die" is actually a "Fair Die" just because the averages match.
• The Consequence: The authors prove mathematically that this shortcut is always worse than the full Bayesian method whenever there is any uncertainty left. It's like driving a car with your eyes closed, guessing the road is straight, when it might actually be curving. You will eventually crash (make a bad prediction).

The "Hill's A(n)" Exception (The Good News)

The paper isn't all bad news. It highlights a specific, famous method called Hill's A(n) (based on the Jeffreys prior).

• This method is special because it naturally fills in all the missing "shape" information.
• It's like having a magical compass that not only points North (the average) but also tells you exactly how much the fog is swirling.
• Because it has this full picture, it works perfectly for predicting 1 step, 2 steps, or 100 steps ahead.

Summary of Key Takeaways

1. One Step is Easy: If you only care about the next event, knowing the average probability is enough.
2. Two Steps is Hard: If you care about a sequence of events, the average is not enough. You need to know how "uncertain" you are (the variance).
3. The Martingale Trap: A system that only promises to keep its "average" consistent (a Martingale) is under-determined. It leaves the future multi-step predictions ambiguous.
4. Don't Cheat: Using the current average as a fixed truth (Plug-in) is mathematically proven to be a bad strategy compared to using the full distribution.
5. The Solution: To predict the future perfectly, you need the full map of your uncertainty (the full conditional law), not just the center point.

In a nutshell: You can't predict a long journey just by knowing the starting direction. You need to know how bumpy the road is, too. The paper tells us that many modern statistical shortcuts only give us the direction, leaving us blind to the bumps.

Technical Summary

Here is a detailed technical summary of the paper "Predictive Coherence and the Moment Hierarchy: Martingale Posteriors for Exchangeable Bernoulli Sequences" by Nicholas G. Polson and Daniel Zantedeschi.

1. Problem Statement

The paper investigates the limitations of Martingale Posteriors, a framework introduced by Fong, Holmes, and Walker (2023) as a likelihood-free alternative to Bayesian inference. In this framework, the posterior sequence $(\theta_n)$ is defined solely by a martingale coherence condition:
$$E[\theta_n \mid \mathcal{F}_{n-1}] = \theta_{n-1} \quad \text{a.s.}$$
where $\theta_n$ represents the posterior mean of the directing parameter $\theta$ given data $\mathcal{F}_n = \sigma(X_{1:n})$.

The Core Question: Does specifying only the first conditional moment (the mean) of the terminal value $\theta_\infty$ uniquely determine multi-step predictive distributions (e.g., $P(X_{n+1} = \dots = X_{n+k} = 0 \mid \mathcal{F}_n)$ for $k \geq 2$)?

The authors argue that while the martingale condition ensures coherence for one-step predictions ($k=1$), it is insufficient for $k \geq 2$. The paper seeks to characterize the structural gap between "mean-only" coherence and full predictive completeness.

2. Methodology and Theoretical Framework

The authors utilize the Exchangeable Bernoulli setting, where by de Finetti's theorem, observations are conditionally i.i.d. given a mixing measure $\Pi$ on $[0,1]$.

• Moment Hierarchy: They establish that the $k$-step predictive probability of a run of zeros, $P(X_{n+1} = \dots = X_{n+k} = 0 \mid \mathcal{F}_n)$, is equal to the posterior expectation $E[(1-\theta)^k \mid \mathcal{F}_n]$.
• Binomial Expansion: Using the binomial theorem, they show:
  $$E[(1-\theta)^k \mid \mathcal{F}_n] = \sum_{j=0}^k \binom{k}{j} (-1)^j E[\theta^j \mid \mathcal{F}_n]$$
  This reveals that the $k$-step predictive depends on all posterior moments up to order $k$.
• Sanov's Theorem & KL Geometry: The paper connects the posterior shape to the Kullback-Leibler (KL) divergence rate function. The martingale condition fixes the location (mean) of the posterior but leaves the curvature (variance and higher moments) undetermined.
• Hausdorff Moment Problem: The authors leverage the fact that on the compact interval $[0,1]$, a probability measure is uniquely determined by its moment sequence (Hausdorff determinacy). This allows them to prove that if the moments are not fixed, the measure (and thus the predictive distribution) is not unique.

3. Key Contributions and Results

A. The Moment Hierarchy and Insufficiency Theorem (Theorem 6.3)

The central result is that the mapping from the posterior mean ($m_n$) to the $k$-step predictive probability is set-valued for $k \geq 2$.

• Non-Uniqueness: Two distinct posterior distributions can share the same mean $m_n$ but yield different values for $E[(1-\theta)^k \mid \mathcal{F}_n]$.
• Implication: The martingale condition (1) alone does not uniquely identify multi-step predictives. It constrains only the first moment, leaving the conditional law of $\theta_\infty$ underdetermined.

B. Quantitative Discrepancy and Plug-in Dominance (Proposition 7.3)

The paper quantifies the error introduced by using a "plug-in" predictor (using the mean $\theta_n$ directly) versus the true Bayes predictive.

• Variance Gap: For $k=2$, the discrepancy is exactly the posterior variance:
  $$E[(1-\theta)^2 \mid \mathcal{F}_n] - (1-\theta_n)^2 = \text{Var}(\theta \mid \mathcal{F}_n)$$
• Strict Domination: Under any strictly proper scoring rule (e.g., Log Score, Brier Score), the plug-in rule is strictly dominated by the Bayes predictive whenever the posterior is non-degenerate (variance $>0$). The plug-in rule systematically underestimates the probability of runs (due to Jensen's inequality on the convex function $(1-\theta)^k$).

C. The Closure Theorem (Theorem 10.3)

The authors establish a necessary and sufficient condition for Predictive Completeness:

• A martingale posterior is predictively complete (uniquely determines all $k$-step predictives) if and only if the conditional law of the terminal value $\theta_\infty$ given $\mathcal{F}_n$ is uniquely specified.
• On $[0,1]$, specifying the law is equivalent to specifying the entire sequence of conditional moments.

D. Positive Example: Hill's $A(n)$ Rule (Section 8)

The paper provides a constructive example where predictive completeness is achieved: Hill's $A(n)$ rule under the Jeffreys prior ($\text{Beta}(1/2, 1/2)$).

• In this specific case, the update rule implicitly specifies the full conditional law (Beta distribution), thereby determining all higher-order moments and satisfying predictive completeness.
• Numerical examples show that the relative gap between plug-in and Bayes predictions grows significantly with the horizon $k$ (e.g., from 9.3% at $k=2$ to 37.8% at $k=4$ in the provided example).

4. Significance and Implications

1. Limitations of Mean-Only Inference: The paper rigorously demonstrates that "mean-calibrated" updating (martingale posteriors) is insufficient for sequential decision-making involving multi-step horizons. Practitioners relying solely on the martingale condition without specifying a likelihood or full conditional law face an inherent ambiguity in predicting future blocks of data.
2. Decision-Theoretic Consequences: The inadmissibility of plug-in rules for $k \geq 2$ implies that ignoring posterior uncertainty (variance) leads to suboptimal decisions and strictly higher expected loss under proper scoring rules.
3. Structural Requirements for Coherence: The work clarifies that to achieve full predictive coherence in exchangeable sequences, one must either:
   • Specify a full prior and likelihood (Classical Bayesian).
   • Specify the full conditional law of the directing parameter (as in specific Martingale Posterior implementations like Hill's $A(n)$).
   • Explicitly constrain higher-order moments (Goldstein's prevision framework).
4. Connection to Information Theory: The results link the insufficiency of first-moment constraints to the geometry of the Sanov rate function (KL divergence). The first moment fixes the linear approximation (location), but multi-step predictives require the quadratic and higher-order curvature terms (variance and cumulants).

5. Conclusion

Polson and Zantedeschi conclude that while the martingale posterior framework offers a flexible, likelihood-free approach to updating beliefs, it suffers from a structural obstruction regarding predictive completeness. The first-moment coherence condition is necessary but not sufficient for multi-step prediction. To resolve this, the framework must be augmented to uniquely specify the conditional law of the directing parameter, effectively recovering the full moment hierarchy required for accurate $k$-step forecasting.