Non-parametric finite-sample credible intervals with one-dimensional priors: a middle ground between Bayesian and frequentist intervals

This paper proposes a new class of non-parametric, finite-sample statistical intervals that occupy a middle ground between Bayesian and frequentist approaches by utilizing one-dimensional priors to achieve credible intervals that require assigning at least p% belief after observing the interval but before inspecting the full dataset.

The Gist

The Big Problem: The "Too Hard" vs. "Too Rigid" Dilemma

Imagine you are trying to guess the average height of everyone in a massive, mysterious city. You have a sample of people, but you don't know the rules of the city (the distribution). You need to give a range (an interval) where you think the true average height lies, and you want to be 95% sure you are right.

You have two traditional options, but both have flaws:

1. The "Bayesian" Approach (The Crystal Ball):

   • How it works: You start with a strong guess (a "prior") about what the city looks like. You combine your guess with the data you collected.
   • The Good: Once you see the data, you can say, "I am 95% sure the answer is in this box." It feels very natural for decision-making.
   • The Bad: To do this properly in a complex, non-parametric world (where you don't know the shape of the data), you have to guess the entire universe of possibilities. It's like trying to predict the weather by guessing the exact position of every single water molecule in the atmosphere. It's mathematically impossible and practically too hard.

2. The "Frequentist" Approach (The Rigid Machine):

   • How it works: You ignore your gut feelings and use a strict formula based only on the data.
   • The Good: It's objective. Everyone gets the same answer.
   • The Bad: The "95% confidence" it gives you is a trick of the long run. It means "If we did this experiment 100 times, 95 of the resulting boxes would catch the truth." But for this specific box you are holding right now? You can't actually say you are 95% sure the answer is inside it. In fact, sometimes you might know for a fact the answer is outside the box, but the machine still says "95% confidence." This makes it useless for making real-life decisions.

The Solution: The "Middle-Ground" Interval

The author, Tim Ritmeester, proposes a new type of interval that sits right in the middle. Think of it as a "Trust-But-Verify" approach.

The Core Idea:
Instead of needing to know the entire shape of the universe (like the Bayesian) or ignoring your gut feelings entirely (like the Frequentist), this new method asks for a very simple, one-dimensional guess: "What is your prior belief about the specific number we are trying to find?"

• The Analogy: Imagine you are betting on a horse race.
  • Bayesian: You need to know the breeding history, diet, and shoe size of every horse in the world to place a bet. (Too hard).
  • Frequentist: You just look at the track conditions and ignore the horses entirely. You get a result, but you can't really trust it for this specific race.
  • The New Method: You just need to say, "I think this specific horse has a 50/50 chance of winning." You don't need to know about the other horses. Based on that simple belief and the race data, the method gives you a betting range.

How It Works (The "Black Box" Trick)

The paper introduces a clever rule for these new intervals:

"After you see the interval (the box), but before you peek inside the raw data yourself, you should be at least 95% sure the answer is in there."

This is a subtle but powerful shift.

• Frequentist: You are 95% sure before you even run the experiment.
• Bayesian: You are 95% sure after you see everything.
• New Method: You are 95% sure after you see the result (the box), provided you haven't peeked at the raw data yet.

Why is this useful?
In the real world, we often see the result (the interval) before we have time to analyze the raw data. This method guarantees that the result is trustworthy at that specific moment.

The Two Specific Cases

The author tested this idea on two common problems:

1. The "Below the Line" Problem (CDF):

   • Question: What percentage of people are shorter than 5'10"?
   • Result: The new method works perfectly. As you get more data, it becomes just as accurate as the best Bayesian method, but it's much easier to calculate.

2. The "Average Height" Problem (Mean):

   • Question: What is the average height of the city?
   • Result: The new method is slightly "wider" (more cautious) than the perfect Bayesian method. It's like wearing a slightly larger safety helmet. It's not as tight as the Bayesian one, but it's much more reliable than the Frequentist one for small amounts of data.

Why Should You Care? (The Benefits)

1. No "God Mode" Required: You don't need to be a genius to guess the shape of the entire universe. You just need a simple guess about the number you are looking for.
2. Flexible: You can change your mind about your "prior guess" (your gut feeling) without breaking the math. You can also add new data as it comes in (sequential sampling) easily.
3. Small Data Superpower: If you only have a few data points (a small sample), this method gives you a much tighter, more useful range than the rigid Frequentist methods.
4. Decision Ready: Unlike Frequentist methods, you can actually say, "I am 95% confident this is the answer," which is exactly what you need when making business or policy decisions.

The Bottom Line

This paper offers a practical compromise. It admits that we can't always know everything (the Bayesian dream) but refuses to accept answers we can't trust (the Frequentist flaw).

It gives us a tool that is easy to use (only needs a simple prior), trustworthy (gives real probabilities), and flexible (works well with small data). It's the "Goldilocks" interval: not too hard to calculate, not too rigid, and just right for making decisions in an uncertain world.

Technical Summary

1. Problem Statement

The paper addresses the limitations inherent in the two dominant paradigms of statistical inference: Bayesian and Frequentist methods.

• Bayesian Credible Intervals: While they allow for a direct probabilistic interpretation (assigning a $p\%$ belief that the parameter lies in the interval after observing data), they require specifying a prior distribution over the entire parameter space. In non-parametric settings, this requires a high-dimensional prior over the space of all possible distributions, which is often impractical, subjective, and computationally complex.
• Frequentist Confidence Intervals: These are objective and do not require priors. However, their interpretation is strictly pre-data: one can only claim a $p\%$ success rate before seeing the data or the interval. Once the data and interval are observed, one cannot generally assign a $p\%$ belief that the specific interval contains the true parameter (in some cases, one can be certain it does not). This makes them problematic for decision-making under uncertainty.

The author proposes a "middle ground" approach that retains the finite-sample interpretability of Bayesian intervals (assigning belief after seeing the interval) while avoiding the complexity of high-dimensional non-parametric priors.

2. Methodology

The core innovation is a relaxation of the definition of a credible set. The proposed method constructs intervals based on a one-dimensional prior over the parameter of interest ($\theta$), rather than a prior over the full distribution space.

The Validity Criterion

A set $S_p$ is defined as a $p\%$ credible set if, after observing the set $s$ (but without inspecting the raw dataset $X$), the user's belief satisfies:
$$b(\theta \in s \mid S_p = s) \geq p$$
This contrasts with standard Bayesian intervals where the belief is conditioned on both the data and the interval ($b(\theta \in s \mid X, S_p=s) \geq p$).

The Construction Algorithm

The method relies on a functional $m = M(X)$ that summarizes the data. The algorithm computes a function $C(s)$ representing a lower bound on the belief $b(\theta \in s \mid m)$. If $C(s) \geq p$, the interval is valid. The general form of the interval is:
$$p \leq \frac{\int_{S_p} l(\theta)b(\theta) d\theta}{\int_{-\infty}^{\infty} l(\theta)b(\theta) d\theta}$$
where $b(\theta)$ is the user's one-dimensional prior and $l(\theta)$ is a likelihood-like function derived from the data summary $m$.

The paper derives concrete implementations for two non-parametric cases:

1. CDF Estimation ($\theta = P(X < y)$):

   • Data Summary: $m$ is the count of samples smaller than $y$ (a binomial variable).
   • Likelihood: $l(\theta)$ is the standard Binomial likelihood: $\binom{N}{m}\theta^m(1-\theta)^{N-m}$.
   • Result: The resulting intervals satisfy the validity criterion with equality.

2. Mean Estimation ($\theta = E[X]$) with Bounded Support:

   • Data Summary: $m = \hat{\mu} + Z$, where $\hat{\mu}$ is the sample mean and $Z$ is a random variable uniformly distributed in $[-\delta, \delta]$. The parameter $\delta$ is tuned based on sample size $N$ and confidence level $p$.
   • Likelihood: Since the exact distribution of $m$ is unknown, the author uses Hoeffding's inequality to derive upper and lower bounds for the probability density, constructing a piecewise likelihood function $l(\mu)$.
   • Result: The intervals satisfy the validity criterion with inequality (conservative).

3. Key Contributions

• Novel Definition: Introduces a statistical interval definition that bridges the gap between Bayesian and Frequentist interpretations, specifically allowing for finite-sample belief assignment after observing the interval but before inspecting the raw data.
• Dimensionality Reduction: Demonstrates that valid non-parametric intervals can be constructed using only a one-dimensional prior on the parameter of interest, bypassing the need for complex priors over the space of distributions.
• Concrete Algorithms: Provides explicit, non-parametric algorithms for estimating the CDF and the mean of bounded distributions.
• Theoretical Guarantees: Proves analytically that the intervals satisfy the proposed validity criterion and derives their asymptotic properties.

4. Results

The paper validates the method through analytical derivation and numerical simulation (using ABC rejection sampling).

• Validity:
  • For CDF estimation, the intervals are exactly valid ($b(\theta \in s \mid S_p=s) = p$).
  • For Mean estimation, the intervals are conservative ($b(\theta \in s \mid S_p=s) \geq p$), as verified numerically.
• Precision (Interval Width):
  • Small Samples: The intervals are narrower than frequentist equivalents because they incorporate the user's prior information.
  • Asymptotic Behavior (CDF): The width converges to that of standard frequentist intervals (and fully Bayesian intervals via the Bernstein-von Mises theorem).
  • Asymptotic Behavior (Mean): The intervals are wider than frequentist Hoeffding-based intervals. For $p=0.95$, they are approximately 48.79% wider than Hoeffding intervals and up to 2.06 times wider than the optimal Bayesian interval in the worst-case variance scenario.
• Flexibility: The method supports sequential sampling and post-hoc analysis naturally. Users can explore different priors or interval definitions without invalidating the statistical properties, provided they do not inspect the raw data $X$ directly.

5. Significance and Discussion

The proposed method offers a practical solution for decision-making under uncertainty in non-parametric settings where:

1. A full non-parametric Bayesian prior is too complex or subjective to specify.
2. Standard frequentist intervals are too rigid or lack a direct probabilistic interpretation for stakeholders.

Trade-offs:

• Advantages: Finite-sample credibility, objectivity regarding the full distribution space (only a 1D prior needed), flexibility in sequential analysis, and narrower intervals for small samples compared to frequentist methods.
• Disadvantages: The requirement to not inspect the raw data to maintain the validity of the belief assignment (though the interval itself is public). For mean estimation, the asymptotic intervals are wider than optimal Bayesian intervals, though this gap may be reducible with better noise modeling.

Future Directions:
The author suggests extending this framework to other statistics, optimizing the noise distribution $Z$ for mean estimation to reduce asymptotic width, and exploring combinations with fiducial statistics to justify non-informative priors through problem symmetries, potentially creating fully non-parametric intervals without subjective priors.

In summary, Ritmeester presents a robust framework for "middle-ground" inference that democratizes the interpretability of Bayesian intervals for non-parametric problems while retaining the computational and conceptual simplicity of frequentist methods.