Bayesian bivariate survival estimation

This paper addresses the challenges of nonparametric bivariate survival estimation by demonstrating the inconsistency of Dirichlet process priors and proposing a consistent alternative using a Beta process prior with a specialized updating scheme that avoids negative mass issues.

The Gist

Imagine you are a detective trying to solve a mystery about how long two things last before they "break" or "happen."

In the simple, one-dimensional world (like tracking just one person's lifespan), we have a famous, reliable tool called the Kaplan-Meier estimator. It's like a sturdy, well-oiled machine that takes a pile of messy data (some people died, some were lost to follow-up) and gives you a clear picture of survival rates.

But what happens when you try to track two things at once? Maybe you want to know how long a husband and wife both survive, or how long two different parts of a machine last. This is the Bivariate Survival problem.

The authors of this paper, Ghosh, Hjort, Messan, and Ramamoorthi, are saying: "Hey, the old machine doesn't work here. If you try to force it, it breaks and gives you impossible answers, like saying there's a negative amount of people left alive!"

Here is the story of their paper, broken down into simple concepts.

1. The Problem: The "Negative Mass" Ghost

In the world of statistics, "mass" just means probability or the number of people in a group. It should always be a positive number (0% to 100%).

The paper starts by pointing out that the most popular attempt to fix the two-variable problem (called the Dabrowska estimator) is flawed. It's like a scale that sometimes tells you you have -5 pounds of apples. It's mathematically consistent in some ways, but physically impossible.

Even worse, they show that a popular "Bayesian" approach (using a specific type of prior called a Dirichlet process) is inconsistent.

• The Analogy: Imagine you are trying to guess the average height of a group of people. You have a "prior guess" (a hunch) that everyone is 6 feet tall. As you measure more and more people, a good method should eventually ignore your hunch and tell you the real average (say, 5'8").
• The Failure: The authors prove that with the old Bayesian method, no matter how much data you collect, your estimate gets stuck at a wrong answer. It refuses to learn the truth. It's like a GPS that keeps telling you to turn left even after you've driven 100 miles in the wrong direction.

2. The Solution: The "Beta Process" Toolkit

The authors propose a new way to build a statistical model using something called Beta Processes.

Think of the survival of two items (like a husband and wife) as a complex dance. To understand the dance, the authors break it down into three simpler steps:

1. Who dies first? (Or do they die at the exact same time?)
2. If one dies first, how long does the other survive after that?
3. What are the odds of them dying together?

They realized that the bivariate problem is actually just a chain of one-dimensional problems (the kind we already know how to solve) linked together.

3. The Trick: Ignoring the "Noise"

Here is the clever part. When you look at the data, there is a lot of "noise"—information that is technically there but doesn't actually help you figure out the survival curve.

• The Metaphor: Imagine you are trying to hear a singer in a noisy room. The old methods tried to record every sound in the room (the singer, the clinking glasses, the traffic outside) and then mathematically subtract the noise. This was too hard and led to errors.
• The New Approach: The authors say, "Let's just ignore the clinking glasses and the traffic." They propose using an "incomplete likelihood." They only use the parts of the data that clearly tell us about the survival times (the singer's voice) and throw away the confusing parts.

By ignoring the "noise," they can build a model that:

1. Never gives negative probabilities (no more -5 pounds of apples).
2. Is Consistent (as you get more data, it gets closer and closer to the truth).
3. Is mathematically clean (it uses the Beta process, which is a flexible, friendly tool for this job).

4. The Result: A Better Map

In the final section, they test their new method against the old, broken Dabrowska method using a made-up example.

• Old Method: Gives a map where the probability of surviving past a certain point is higher than the probability of surviving past an earlier point. This is illogical (like saying you are more likely to survive to age 80 than age 70).
• New Method: Produces a smooth, logical map that makes sense. It respects the rules of probability and gives a realistic picture of how the two items survive together.

Summary

This paper is about fixing a broken tool.

• The Problem: Trying to track two lifetimes at once with old methods leads to impossible math (negative numbers) and stubborn errors (inconsistency).
• The Fix: The authors built a new, modular tool using Beta Processes.
• The Secret Sauce: They simplified the math by ignoring the confusing parts of the data that don't actually help, allowing them to create a model that is both logical and accurate.

It's a reminder that sometimes, to solve a complex problem, you don't need to use more data; you need to use the right data and a smarter way of looking at it.

Technical Summary

1. Problem Statement

The paper addresses the challenge of nonparametrically estimating bivariate survival distributions (the joint distribution of two survival times, $T_1$ and $T_2$) in the presence of right-censoring.

• The Difficulty: Unlike the univariate case, where the Kaplan–Meier and Nelson–Aalen estimators are standard, extending these to the bivariate case is non-trivial.
• Existing Limitations:
  • Dabrowska (1988) Estimator: While consistent, it is not a proper survival distribution because it can assign negative mass to certain subsets of the sample space.
  • Bayesian Approaches (Dirichlet Process): Previous attempts using Dirichlet process priors (e.g., Pruitt, 1988) suffer from inconsistency. The posterior distribution does not converge to the true distribution even with infinite data.
• Goal: To develop a Bayesian nonparametric framework for bivariate survival that avoids negative mass and ensures posterior consistency.

2. Methodology

The authors propose a novel approach based on Beta processes and a specific reparametrization of the bivariate data structure.

A. Reparametrization of the Bivariate Model

The core of the methodology involves decomposing the bivariate survival time vector $\mathbf{T} = (T_1, T_2)$ into a sequence of one-dimensional components. This is achieved by defining:

1. Minimum Time: $T^* = T_1 \wedge T_2$.
2. Order Indicator ($\epsilon$): A variable indicating the relationship between $T_1$ and $T_2$:
   • $\epsilon = 0$ if $T_1 = T_2$
   • $\epsilon = 1$ if $T_1 > T_2$
   • $\epsilon = 2$ if $T_1 < T_2$
3. Conditional Distributions:
   • Distribution of $T_1$ given $T^*$ and $\epsilon=1$.
   • Distribution of $T_2$ given $T^*$ and $\epsilon=2$.

Similarly, the observed censored data $(\mathbf{Z}, \mathbf{\Delta})$ is reparametrized into:

• $Z^* = Z_1 \wedge Z_2$ (observed minimum).
• $\Delta^* = I\{T^* \leq C^*\}$ (indicator that the minimum event was observed).
• $\eta$: An indicator analogous to $\epsilon$ but based on observed censored times.

B. The Likelihood and "Incomplete" Likelihood

The authors demonstrate that the mapping from the true distribution $F$ to the distribution of observed data is one-to-one but not onto. This means many empirical distributions of observed data do not correspond to any valid underlying survival distribution.

Consequently, the full likelihood function contains complex terms related to censored observations where $\Delta^* = 0$ (i.e., both components are censored). The authors propose an "essentially Bayesian" approach:

• They ignore the likelihood terms associated with $\Delta^* = 0$.
• They utilize only the "relevant" parts of the likelihood involving:
  1. The distribution of $(Z^*, \Delta^*)$.
  2. The conditional distribution of $\eta$ given $\Delta^*=1$.
  3. The conditional distribution of the specific censored times given $\Delta^*=1$ and $\eta$.

This simplification allows for a tractable posterior derivation while retaining the essential information needed for estimation.

C. The Prior: Bivariate Beta Process

The authors construct a Bivariate Beta Process prior, which is a natural generalization of the univariate Beta process (Hjort, 1990). The prior is defined on the space of distributions via independent components:

1. $T^*$: Modeled as a Beta process.
2. $\epsilon | T^*$: Modeled as a Dirichlet process.
3. $T_1 | (T^*, \epsilon=1)$ and $T_2 | (T^*, \epsilon=2)$: Modeled as independent univariate Beta processes.

3. Key Contributions

1. Proof of Inconsistency for Dirichlet Priors: The paper provides a simplified, rigorous proof that the Dirichlet process prior leads to an inconsistent posterior for bivariate survival estimation (extending Pruitt's earlier work). They show that under specific true distributions, the Bayes estimator converges to a mixture of the true distribution and the prior guess, rather than the true distribution itself.
2. Construction of a Consistent Estimator: By utilizing the Bivariate Beta process and the incomplete likelihood strategy, the authors derive a posterior distribution that is consistent.
3. Avoidance of Negative Mass: Unlike the Dabrowska estimator, the proposed Bayesian estimator is a proper survival distribution, guaranteeing non-negative mass on all subsets.
4. Noninformative Limit: The authors derive a "noninformative" Bayes estimator (by letting prior parameters approach zero) which serves as a frequentist-like estimator with desirable properties.

4. Results

• Posterior Consistency: The paper proves that the posterior distribution derived from the Bivariate Beta process prior, using the incomplete likelihood, converges to the true distribution as sample size $n \to \infty$.
• Explicit Estimators: The authors provide explicit formulas for the Bayes estimators of the hazard functions and survival probabilities.
  • For the discrete case, the posterior parameters are updated by adding observed counts (events) to the prior parameters.
  • The noninformative estimator takes a form analogous to the Kaplan–Meier estimator but applied to the decomposed components ($T^*$, $\epsilon$, and conditional $T_j$).
• Numerical Example: The paper compares the proposed noninformative estimator with the Dabrowska estimator on a small dataset.
  • Dabrowska: Produced negative mass (e.g., $P(T_1 > x, T_2 > y)$ was larger for a smaller set than for a larger set, violating monotonicity).
  • Proposed Method: Produced a valid, monotonic survival function with no negative mass.

5. Significance

• Theoretical Resolution: The paper resolves a long-standing issue in bivariate survival analysis by demonstrating that standard Bayesian nonparametric tools (Dirichlet processes) fail in this context and providing a viable alternative (Beta processes).
• Practical Applicability: The proposed estimator is computationally feasible and guarantees the fundamental property of survival functions (non-negativity and monotonicity), which is critical for applications in medicine (e.g., husband/wife survival) and engineering (e.g., system component lifetimes).
• Methodological Insight: The strategy of using an "incomplete" likelihood to bypass the non-onto nature of the censoring map in higher dimensions offers a new perspective for handling complex censored data problems where full likelihoods are intractable or ill-posed.

In summary, the paper successfully bridges the gap between Bayesian nonparametrics and bivariate survival analysis, offering a consistent, mathematically sound, and practically useful estimator that overcomes the defects of previous methods.