📊 statistics

From Risk Sets to Martingales: A Counting-Process Framework for Event-History Learning

This paper establishes a unified event-history learning framework based on counting-process notation and martingale theory that translates diverse censored, recurrent, and multistate data problems into five recurring mathematical objects, thereby providing common computational algorithms, theoretical proof templates, and a shared language for deriving and comparing classical and machine-learning survival methods.

Original authors: Eliuvish Han Cui

Published 2026-06-23

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Eliuvish Han Cui

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict when a lightbulb will burn out, or when a patient will relapse after treatment. In the real world, you rarely get to see the exact moment the "event" happens. Sometimes the lightbulb is still working when you stop watching (censoring). Sometimes you only check the bulb once a week and know it broke sometime between Tuesday and Friday (interval censoring). Sometimes a patient can get sick, get better, get sick again, and then die (multistate).

This paper is essentially a universal translator and a construction kit for handling all these messy, incomplete timelines. The author, Elvis Han Cui, argues that instead of treating every type of survival data as a unique, confusing problem, we can view them all through a single, powerful mathematical lens called the Counting-Process Framework.

Here is the breakdown of the paper's ideas using everyday analogies:

1. The Core Idea: The "Risk Set" and the "Surprise"

The paper starts with a simple equation: $dN(t) = Y(t)\alpha(t)dt + dM(t)$ .
Think of this as a recipe for predicting the future:

$Y(t)$ (The Risk Set): This is the crowd of people or things currently "in the game." If you are studying lightbulbs, $Y(t)$ is the number of bulbs still on. If a bulb burns out or you stop watching it, it leaves the crowd.
$\alpha(t)$ (The Hazard): This is the "instantaneous pressure" or risk of failure at this exact moment.
$dN(t)$ (The Jump): This is the actual event happening (a bulb burning out, a patient relapsing).
$dM(t)$ (The Martingale/Surprise): This is the most important part. It represents the unexpected. Even if you know the crowd size and the risk, you can't predict exactly when the next bulb will blow. The "Martingale" is the mathematical way of saying, "The difference between what we expected to happen and what actually happened is just random noise."

The paper's main claim is that if you can separate the predictable crowd from the random surprise, you can build a unified system to analyze almost any type of time-to-event data.

2. The Five Building Blocks

Instead of memorizing hundreds of different formulas for different scenarios, the paper says every problem can be broken down into five recurring objects:

Risk Process: Who is still in the game?
Jump Process: When did the event actually happen?
Compensator: What did we expect to happen based on the rules?
Estimating Equation: The math tool used to find the answer.
Limiting Argument: The proof that the answer gets better as you get more data.

3. What This Framework Can Do

The paper shows how this single framework handles a wide variety of complex situations, translating them all into the same language:

Standard Survival (Right-Censored): The classic "Kaplan-Meier" curve. Imagine a line of runners. Some drop out (censored) before the finish line. The framework calculates the probability of finishing by looking at who is still running at every step.
Multiple States (Multistate): Imagine a patient who goes from "Healthy" $\to$ "Sick" $\to$ "Dead," or "Healthy" $\to$ "Dead" directly. The paper uses a product integral (like multiplying a series of small probabilities) to track the probability of being in any specific state at any time. It's like a flowchart where you multiply the chances of taking each path.
Recurrent Events: What if a patient gets sick, gets better, and gets sick again? The framework counts every "jump" (sickness) as long as the patient is still in the risk set.
Interval Censoring: Imagine you only check a patient's health on Mondays and Wednesdays. If they are sick on Wednesday but healthy on Monday, you know the event happened in between, but not exactly when. The paper uses a "self-consistency" algorithm (like an EM algorithm) to redistribute the probability mass across the possible time intervals until it finds the best fit.
Causal Inference (Instrumental Variables): Sometimes, a treatment (like a drug) is given to sicker patients, making it look like the drug is bad. To fix this, the paper uses "Instrumental Variables" (like a genetic marker or a doctor's random preference) to isolate the true causal effect, separating the drug's effect from the patient's underlying health.

4. The "Cross-Fitted" Innovation

One of the paper's new contributions is a method for validating modern machine learning models (like neural networks) used for survival data.

The Problem: If you train a model on a dataset and test it on the same data, it might just be memorizing the answers (overfitting).
The Solution: The paper proposes a "cross-fitted" approach. You train the model on 90% of the data and test it on the remaining 10%.
The Magic: The paper proves a new mathematical identity: if your model is good, the "surprise" (martingale residuals) in the test group should look like random noise. If the model is bad, the "surprise" will show a pattern. This gives a rigorous way to check if a complex AI model is actually learning the event history or just guessing.

5. Why This Matters

The paper doesn't just list old methods; it provides a common language.

For Mathematicians: It gives "proof templates." Instead of proving a new theorem from scratch for every new type of data, you can plug your problem into these five building blocks and use the existing proofs.
For Data Scientists: It turns complex survival analysis into reusable algorithms. Whether you are analyzing lightbulbs, historical office tenures, or patient relapses, the "risk-set sweep" (Algorithm 1) is the same.
For Everyone: It unifies the field. Whether you are doing a simple "Log-Rank test" to compare two groups or a complex "Bayesian nonparametric" model for multistate transitions, they are all just different ways of managing the Risk Set and the Surprise.

Summary

Think of this paper as a universal adapter for time-to-event data. Whether your data is messy, incomplete, has multiple states, or involves causal questions, the author shows that you can strip away the complexity and look at the core mechanics: Who is at risk? What did we expect? What actually happened? And was the difference just random noise? By answering these questions consistently, we can derive, check, and compare any method for analyzing event histories.

Technical Summary: From Risk Sets to Martingales: A Counting-Process Framework for Event-History Learning

Problem Statement

Survival and event-history data are intrinsically filtered observations rather than complete response variables. At any time $t$ , the analyst observes the accumulated history (risk sets, covariates, prior states) but only observes the event jump when it occurs. This temporal asymmetry complicates the analysis of censored, truncated, recurrent, multistate, and covariate-dependent data. While numerous specific methods exist (e.g., Kaplan–Meier, Cox regression, Fine–Gray), there is a need for a unified framework that separates predictable risk-set information from observed event jumps to facilitate the derivation, checking, and comparison of both classical and machine-learning methods for event-history data.

Methodology

The paper establishes a unified framework based on the counting-process notation and the multiplicative-intensity representation:
$dN(t) = Y(t)\alpha(t)dt + dM(t)$
where:

$N(t)$ is the jump process (observed event count).
$Y(t)$ is the predictable risk process (number at risk).
$\alpha(t)$ is the intensity function.
$Y(t)\alpha(t)dt$ is the compensator increment.
$dM(t)$ is the martingale innovation.

Rather than cataloging specific survival methods, the framework translates every partially observed learning problem into five recurring objects:

Risk Process: The predictable set of subjects eligible for an event.
Jump Process: The observed event increments.
Compensator: The predictable intensity model.
Estimating Equation: Derived from martingale properties or likelihoods.
Limiting Argument: The asymptotic justification (e.g., martingale central limit theorems).

The paper systematically applies this framework across four domains:

Nonparametric Estimation: Deriving Nelson–Aalen (cumulative hazard), Kaplan–Meier (survival), Aalen–Johansen (multistate), Dabrowska (bivariate), and Turnbull (interval-censored) estimators as product-integral updates or likelihood maximizations.
Hypothesis Testing: Formulating log-rank and weighted tests as martingale risk-set contrasts, extending to stratified, crossing-hazard, and competing-risk scenarios.
Regression Modeling: Unifying Cox–Andersen–Gill (multiplicative), Aalen (additive), transformation/AFT, and semi-Markov models. It explicitly handles duration dependence by distinguishing between calendar time and elapsed sojourn time.
Causal Inference: Adapting instrumental variable (IV) methods (e.g., G-estimation, structural Cox models) to censored outcomes by combining causal moment restrictions with counting-process estimating equations.

The framework also integrates modern machine learning by mapping flexible learners (random forests, neural networks) into this structure, specifically addressing cross-fitted validation for censored data.

Key Contributions

1. Computational Unification

The paper converts diverse computational tasks into reusable algorithms and simulation diagnostics:

Risk-set sweeps: A unified algorithm (Algorithm 1) for Nelson–Aalen, Kaplan–Meier, log-rank, and Cox score calculations.
Product-integral updates: Algorithms for multistate transitions (Aalen–Johansen) and semi-Markov elapsed-time bookkeeping.
Interval likelihoods: The Turnbull self-consistency/EM-ICM blueprint for interval-censored data.
Cross-fitted validation: A wrapper (Algorithm 4) for applying modern learners to censored data while respecting the predictable risk set and filtration.

2. Theoretical Contributions

Proof Templates: The paper provides a standardized structure for proving consistency, asymptotic normality, and variance estimation for a wide class of estimators using martingale decompositions and functional delta methods.
New Theoretical Result (Theorem 27): The paper proves a new out-of-fold compensator validation identity for cross-fitted censored learners. It establishes that conditional on training folds, validation-fold residual scores are martingale-centered if and only if the learned intensity is calibrated in a weighted integrated-intensity sense. This interprets cross-fitting as a filtration requirement: validation predictions may depend on training data and pre- $t$ history but not on future validation jumps.

3. Application Mapping

The framework maps diverse domains—biomedical, reliability, operational, economic, financial, literary, historical, and causal survival examples—onto the same risk-set and compensator language. This allows for a common mathematical language to derive and compare methods across these fields.

Results and Scope

The paper does not present new empirical data but rather a theoretical and algorithmic synthesis. Its "results" are the successful unification of:

Estimators: From simple right-censored curves to complex bivariate interval-censored surfaces and multistate transition probabilities.
Tests: From standard log-rank tests to adaptive, weighted, and competing-risk comparisons.
Models: From parametric (Weibull, Gompertz) to semiparametric (Cox, Aalen) and fully nonparametric (Beran, Turnbull) models.
Causal Methods: Structural Cox and additive-risk IV models that handle endogeneity in time-to-event outcomes.

The paper demonstrates that the counting-process decomposition $dN = Y\alpha dt + dM$ serves as the "common analytic basis" for all these methods. It shows that the Nelson–Aalen estimator is the prototype for cumulative hazard estimation, the Kaplan–Meier estimator is a product-integral transformation of it, and the Aalen–Johansen estimator is the multistate extension of the same logic.

Significance and Claims

The paper claims to provide a common mathematical language for deriving, checking, and comparing classical and machine-learning methods for censored, recurrent, and multistate event-history data.

For Classical Analysis: It offers proof templates that clarify the recurring martingale, likelihood, and product-integral arguments, making the derivation of new estimators more systematic.
For Modern Learning: It addresses the integration of flexible machine learning models (e.g., neural networks, random forests) with the strict requirements of censored data inference. Specifically, it provides the theoretical justification (Theorem 27) for using cross-fitting in survival settings, ensuring that validation residuals behave as martingales only when the model is properly calibrated.
For Causal Inference: It bridges the gap between instrumental variable theory and survival analysis, showing how to construct valid G-estimators and structural models that respect the filtration of censored data.

The authors position this work not as a catalogue of named methods, but as an organizing framework where the "risk set, censoring mechanism, and filtration restrictions" are made explicit for any learning problem. The ultimate goal is to facilitate the transfer of methods across domains (e.g., from clinical trials to reliability engineering) by ensuring they share the same underlying stochastic structure.