Either a Confidence Interval Covers, or It Doesn't (Or Does It?): A Model-Based View of Ex-Post Coverage Probability

Here is an explanation of the paper using simple language, creative analogies, and metaphors.

The Big Question: "Did the Net Catch the Fish?"

Imagine you are a fisherman. You have a net that, over the long run, catches fish 95% of the time. This is your "Confidence Interval."

One day, you cast your net. You pull it in. There is a fish inside.

The Traditional View (Neyman's Slogan): A strict statistician would say, "Stop talking about probability! The fish is either in the net or it isn't. Since you can see the fish, the probability is 100% (or 0%). There is no 'maybe' left. Probability only exists before you cast the net."
The Author's View (Scott Lee): This paper argues that the strict view is too rigid and actually breaks the logic of statistics. Just because you saw the fish doesn't mean you should throw away the math that tells you your net is usually good. We can still talk about the "chance" that this specific catch was a success, even after we've seen it.

The Core Argument: Two Ways to Look at the Same Event

The author says there are two valid ways to look at probability, and we shouldn't ban one just because we've seen the result.

1. The "Design Level" View (The Blueprint)

Think of a factory making toy cars. The blueprint says: "95% of these cars will have working brakes."

Before you build: You say, "There is a 95% chance this car will have working brakes."
After you build: You look at the car. It has working brakes.
The Strict Statistician says: "Now that you see the brakes work, the probability is 100%. The 95% number is useless for this specific car."
The Author says: "Wait a minute. If I have to decide whether to sell this car to a customer, I still want to know: 'Is this car part of the 95% that works, or the 5% that fails?' Even though I can't see the internal mechanism right now, the process that made it still has a 95% success rate. Ignoring that rate makes me a worse decision-maker."

2. The "Microstate" View (The Infinite Movie)

Imagine a movie camera recording an infinite number of fish casts.

The Strict View: In any single frame of the movie, the fish is either in the net or not. It's a fact.
The Author's View: The "probability" isn't about the single frame; it's about the pattern of the whole movie. Even if we are looking at one specific frame where the fish is caught, that frame is part of a long sequence where 95% of the frames show a catch. We can still talk about the "likelihood" of this frame belonging to the "catch" group based on how the camera (the experiment) was set up.

The Thought Experiments (Why the Strict View is Silly)

The author uses three fun stories to show why saying "it's either 0 or 1" causes real-world problems.

Story 1: The Sick Patient (The Doctor)

A patient tests positive for the flu. The test is 81% accurate (Positive Predictive Value).

Strict Logic: "The patient either has the flu or doesn't. Since we don't know which, we can't assign a probability. We just have to guess blindly."
The Problem: If the doctor follows this, they can't use the 81% number to decide whether to prescribe medicine. They would have to say, "I don't know if she has it, so I won't treat her." That's bad medicine!
The Fix: We must use the 81% number to make a decision, even though the patient's true health status is already fixed.

Story 2: The Cat and the Treats

A cat, Sophie, gets a treat. 75% are seafood (she loves them), 25% are chicken.

Scenario: The owner picks a treat, but doesn't look at it. The cat eats it and purrs.
Strict Logic: "The treat was either seafood or chicken. Once the cat ate it, the flavor was fixed. We can't talk about the probability of it being seafood anymore."
The Problem: If the owner wants to know, "What are the odds that treat was seafood?" they need to use the math. If they refuse to use the math because "the flavor is already decided," they can't predict the cat's behavior or understand the situation.

Story 3: The Chocolate Truffles (The Machine)

A machine makes chocolate truffles. Sometimes it fails to fill them (they are hollow). A second machine checks them.

Scenario: The first machine makes a truffle. The second machine hasn't checked it yet.
The Puzzle: If we say "The truffle is either filled or hollow," we create a paradox. If we assume it's filled, the math says the next truffle has a 90.5% chance of being filled. If we assume it's hollow, the math says the next truffle has a 90% chance.
The Author's Point: If we force ourselves to pick one "true" state (filled or hollow) before we know it, we break the math that predicts the next truffle. We need to keep the "90% chance" alive to make sense of the machine's future behavior.

The Solution: "Predictive Probability"

The author suggests we stop treating probability as something that "dies" once we see the data. Instead, he proposes a new way to think about Confidence:

Confidence is like a Weather Forecast.

Before the storm: The meteorologist says, "There is a 95% chance of rain."
During the storm: It is raining.
The Strict View: "It's raining! The probability is 100%. The forecast was just a guess."
The Author's View: "The forecast was a prediction based on the weather model. Even though it's raining now, the quality of the forecast is still defined by that 95% accuracy. If I want to know if I should have brought an umbrella before I saw the rain, I rely on that 95%."

The New Rule:
When we look at a confidence interval after the fact, we shouldn't just say "It's 0 or 1." We should say:
"Based on the method we used, and the data we have, how likely is it that this specific interval is one of the 'good' ones?"

This allows us to:

Keep the long-run accuracy (the 95% rule).
Make smart decisions in the real world (like the doctor prescribing medicine).
Avoid the logical trap of saying "Probability doesn't exist anymore."

The Takeaway

The paper argues that the famous slogan "Either it covers, or it doesn't" is too restrictive. It treats probability like a light switch that turns off the moment you look at the data.

Instead, the author wants us to treat probability like a map. Even if you are standing at a specific spot on the map (the realized data), the map still tells you how likely you are to be in a "safe zone" based on how the map was drawn. We can talk about the probability of being in the safe zone even after we've arrived, because that probability describes the reliability of the journey, not just the destination.

Here is a detailed technical summary of the paper "Either a Confidence Interval Covers, or It Doesn't (Or Does It?): A Model-Based View of Ex-Post Coverage Probability" by Scott Lee.

1. Problem Statement

The paper addresses a foundational tension in frequentist statistical inference regarding the interpretation of Confidence Intervals (CIs) after data has been observed (ex-post).

The Standard Neyman Interpretation: Jerzy Neyman's original formulation posits that a CI is a procedure with long-run coverage properties ($1-\alpha $). Once a specific interval is realized from a dataset$ X=x $, the parameter$ \theta$ is either inside or outside the interval. Consequently, the probability of coverage for that specific realized interval is degenerate: it is either 0 or 1. Under this strict "behavioristic" view, making any probabilistic statement about the coverage of a single realized interval ex-post is considered conceptually invalid or a fallacy.
The Conflict: This strict interpretation creates a paradox when applied to real-world scenarios involving "occurred-but-unobserved" events (e.g., a patient's actual disease status after a positive test, or a specific truffle's fill status in a manufacturing line). In these cases, practitioners naturally and intuitively assign intermediate probabilities (e.g., Positive Predictive Value) to guide decisions, yet the strict Neyman logic suggests these probabilities should collapse to 0 or 1 because the event has already occurred.
The Core Question: Is the "either-or" reading the only legitimate interpretation of confidence, or does the frequentist mathematical framework itself permit meaningful ex-post probability statements that are not degenerate?

2. Methodology

Lee employs a dual approach, combining informal thought experiments with formal probability theory to deconstruct the "either-or" dogma.

A. Informal Thought Experiments

The author constructs three scenarios to demonstrate the practical and logical difficulties of adhering strictly to the "either-or" rule:

Dr. I-Don't-No (Medical Diagnosis): A patient tests positive for flu. While the true disease state is fixed (0 or 1), the physician must use the Positive Predictive Value (PPV) to guide treatment. Insisting that the probability is 0 or 1 renders the diagnostic test useless for decision-making.
The Cat Tasting Treats: A cat eats a treat of unknown flavor. The owner calculates the probability of the cat napping. If the flavor is fixed but unknown, the probability splits into two degenerate values (conditional on the specific flavor). However, the unconditional probability (80%) is the only useful metric for prediction before the outcome is known.
We're in Deep Truffle Now (Manufacturing): A chocolatier's machine produces truffles with a known fill probability. If the current truffle's status is fixed but unknown, conditioning on it creates a "forked" probability for the next truffle. The author argues that insisting on the degenerate conditional for the current truffle prevents the recovery of the design-level probability for the next truffle, breaking the model's predictive utility.

B. Formal Mathematical Argument

The paper recasts the CI problem using Kolmogorov-style probability theory and the concept of infinite sequences of trials (microstates):

Microstates: The author embeds the CI procedure into an infinite sequence of i.i.d. trials. A "microstate" is a fully fixed infinite sequence of outcomes.
Coverage Indicators: Let $Z_i$ $Z_{i}$ be a Bernoulli indicator variable where $Z_i=1$ $Z_{i} = 1$ if the $i$ $i$ -th interval covers $\theta$ $θ$ .
- Design Level (Ex-ante): $P_\theta(Z_i = 1) = 1-\alpha$ . This is the expectation over the randomness of the data.
- Ex-Post Level: $P_\theta(Z_i = 1 \mid X_i = x_i) = \mathbb{I}(\theta \in I(x_i)) \in \{0, 1\}$ .
The Argument: The author demonstrates that these are not contradictory truths but rather different conditioning levels within the same probability model.
- The design-level probability ($1-\alpha $) corresponds to a coarser$ \sigma$-algebra (the design alone).
- The degenerate probability corresponds to the finest $\sigma$ -algebra (the full data realization).
- The "either-or" slogan privileges the finest conditioning level, but the mathematical machinery (Law of Large Numbers, Borel-Cantelli lemmas) that defines long-run error rates relies on the coarser, non-degenerate design-level probabilities.

3. Key Contributions

Refutation of the "Only Legitimate" Claim: The paper argues that the strict "either-or" interpretation is not the only mathematically valid frequentist interpretation. It is merely a choice of conditioning level (selecting the finest $\sigma$ -algebra) rather than a fundamental property of the model.
Reconciliation of Ex-Post and Ex-Ante: The author shows that frequentist theory can accommodate intermediate ex-post probabilities. By treating the realized interval as one draw from a reference class (the design), one can assign a non-degenerate probability to its coverage status based on the information available, without violating frequentist principles.
Redefinition of "Confidence": The paper proposes reinterpreting "confidence" not just as a long-run frequency, but as a predictive probability or a model-based probabilistic forecast. It is the best guess of a non-oracle observer regarding the coverage of an interval given the available information.
The "Soft Normative Rule": The author proposes a practical guideline for statistical inference: Only condition on post-trial information if it actually reduces uncertainty about the outcome.
- If the realized data provides no additional clues about coverage (e.g., a standard t-interval where the width doesn't reveal $\theta$ ), one should retain the design-level probability ($1-\alpha$) rather than collapsing to 0 or 1.
- Conditioning on the hidden truth (the "maximal" $\sigma$ -algebra) is an epistemic choice, not a mathematical necessity, and often leads to unhelpful degenerate probabilities.

4. Results

Logical Consistency: The thought experiments demonstrate that enforcing the "either-or" rule leads to absurdities in decision-making (e.g., ignoring PPV in medicine) and breaks the continuity of probability models in sequential processes (e.g., the truffle factory).
Mathematical Equivalence: The formal analysis proves that the design-level coverage ($1-\alpha $) and the degenerate conditional (0 or 1) are simply$ E[Z] $and$ E[Z|X]$ respectively. They are linked by the Law of Iterated Expectations. Privileging one over the other is a philosophical stance, not a mathematical constraint.
Reference Class Problem: The paper highlights that the "either-or" view forces the statistician to switch reference classes from the infinite collective defined by the design to a singleton set defined by the realized outcome. This switch is unnecessary and often counter-productive for inference.

5. Significance

Bridging the Gap: This work bridges the gap between the rigid "behavioristic" interpretation of frequentism (Neyman) and the intuitive, epistemic needs of practitioners who must make decisions based on single-case data.
Legitimizing Intermediate Probabilities: It provides a rigorous frequentist justification for making intermediate probability statements about single realized events (e.g., "There is a 95% chance this specific interval covers the parameter") without resorting to Bayesian priors.
Clarifying "Confidence": By framing confidence as a predictive probability, the paper offers a more nuanced understanding of what a CI represents: a forecast of success based on the design, rather than a binary statement about a fixed reality.
Practical Guidance: The proposed "soft rule" offers statisticians a principled way to decide when to report design-level coverage versus when to acknowledge that specific data features might refine (or collapse) that probability, preventing the misuse of the "either-or" slogan to dismiss useful probabilistic reasoning.

In conclusion, Scott Lee argues that the "either-or" slogan is too restrictive. A model-based view allows frequentists to treat the coverage of a realized interval as a random variable with a non-degenerate probability distribution, provided one maintains the appropriate conditioning level relative to the information available. This preserves the utility of frequentist methods for single-case inference while remaining mathematically consistent with the definition of long-run error rates.