📊 statistics

Distribution-Free Selection of Low-Risk Oncology Patients for Survival Beyond a Time Horizon

This paper proposes and compares two distribution-free frameworks—Learn-Then-Test for high-probability guarantees and a hypothesis testing approach for false discovery rate control—to calibrate black-box survival models for selecting low-risk oncology patients, demonstrating a trade-off between statistical power and the strength of theoretical guarantees while addressing right-censored data.

Original authors: Matteo Sesia, Vladimir Svetnik

Published 2026-02-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Matteo Sesia, Vladimir Svetnik

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a doctor running a specialized clinic. You have a limited number of "VIP slots" available for a new, less intensive treatment program. You want to fill these slots with patients who are very likely to stay healthy for the next three months. If you pick someone who gets sick too soon, the program fails, and the patient might be harmed. But if you are too scared to pick anyone, you miss the chance to help people who could benefit.

The problem? You have a "black box" computer program that predicts who is healthy and who isn't. But computers make mistakes, and sometimes they lie about how sure they are. You need a way to calibrate this computer so that you can trust its predictions before you let patients into the program.

This paper introduces two new, mathematically rigorous ways to do this calibration, specifically dealing with the messy reality that some patients drop out of the study early (censoring), so you don't know if they would have gotten sick later.

Here is the breakdown of the paper's ideas using simple analogies:

The Core Problem: The "Guessing Game"

You have a list of 1,000 patients. Your computer says, "Patient A has a 95% chance of being safe," and "Patient B has a 90% chance." You want to pick everyone with a score above a certain line (say, 90%).

But here's the catch: You don't know where to draw the line.

If you draw the line at 99%, you might only pick 5 people. You are safe, but you aren't helping many people.
If you draw the line at 80%, you might pick 500 people, but 20% of them might get sick. That's too risky.

The paper solves the problem of finding the perfect line that lets in as many people as possible while guaranteeing that the "sickness rate" stays below a safe limit (e.g., 10%).

The Two Solutions: The "Safety Net" vs. The "Average"

The authors propose two different philosophies for drawing that line.

1. The "Learn-Then-Test" (LTT) Method: The Safety Net

The Analogy: Imagine you are walking a tightrope. You want to go as far as possible, but you have a safety net below you.

How it works: You test different lines one by one, starting from the strictest (safest) and moving toward the loosest (most inclusive).
The Rule: You keep moving the line until you hit a point where the computer says, "Whoa, there's a chance we might break the safety rule." You stop just before that point.
The Guarantee: This method gives you a High-Probability Guarantee. It's like saying, "I am 95% sure that in this specific instance, we won't have too many sick patients."
The Trade-off: Because it's so worried about being wrong this one time, it tends to be a bit conservative. It might leave some healthy people on the sidelines just to be extra safe.

2. The "FDR" (False Discovery Rate) Method: The Average

The Analogy: Imagine you are a fisherman casting a net into a huge ocean. You can't guarantee that every single fish you catch is the right kind. But you can guarantee that, on average, 90% of your catch is good fish.

How it works: Instead of testing lines one by one, this method treats every patient as a hypothesis: "Is this patient safe?" It uses a special statistical trick (Conformal Prediction) to assign a "suspicion score" to everyone. Then, it uses a famous rule (Benjamini-Hochberg) to decide who to pick.
The Rule: It says, "If we do this experiment 100 times, the average number of sick patients we pick will be under 10%."
The Guarantee: This gives an Expectation Guarantee. It doesn't promise you won't have a bad day; it promises that over the long run, you are safe.
The Trade-off: This method is usually more powerful. It catches more patients (higher yield) because it's willing to take a small risk on a specific day in exchange for a better average performance.

The "Censoring" Problem: The Missing Puzzle Pieces

In real life, some patients leave the study early (maybe they move away or the study ends). You don't know if they got sick or not. This is called "censoring."

The paper uses a clever mathematical trick called IPCW (Inverse Probability of Censoring Weighting).

The Metaphor: Imagine you are counting votes, but some people walked out of the room before voting. To make the count fair, you give a "multiplier" to the votes of the people who stayed. If someone who looks like the people who left stayed, their vote counts for more to represent the missing people.
This allows both methods to work even when data is incomplete.

The Results: Who Wins?

The authors tested these methods on real cancer data and simulated scenarios.

The "Old Way" (Naive): Just trusting the computer's raw numbers usually leads to picking too many risky patients.
The "Super Conservative" Way: Being too scared leads to picking almost no one.
The Winner:
- FDR (The Fisherman): Usually picks the most patients while keeping the average risk low. It's the most efficient.
- LTT (The Tightrope Walker): Picks slightly fewer patients but offers a stronger guarantee that today won't be a disaster.

The Bottom Line for Everyday People

If you are a hospital administrator or a doctor:

If you need to be absolutely sure that this specific group of patients is safe, use the LTT method. It's the cautious, safety-first approach.
If you want to help the maximum number of people and are okay with the risk being low on average over many groups, use the FDR method. It's the efficient, high-yield approach.

The paper essentially gives doctors a new toolkit to stop guessing and start making data-driven decisions that are mathematically proven to be safe, even when the data is messy and incomplete.

1. Problem Statement

The paper addresses the statistical challenge of calibrating a screening rule to select a subset of patients who are unlikely to experience an adverse event (e.g., disease progression or death) within a fixed time horizon $t_0$ . This is critical in clinical settings, such as oncology de-escalation trials, where researchers must enroll only "low-risk" patients (e.g., ensuring $\le 10\%$ event probability within 3 months) to ensure safety while maximizing the number of eligible participants.

Key Challenges:

Black-Box Models: Modern survival models (e.g., Random Survival Forests, Deep Learning) are flexible but lack principled uncertainty quantification. Treating them as "black boxes" requires statistical guarantees at the level of the screening rule, not the model itself.
Right-Censoring: In clinical data, follow-up often ends before an event occurs. Standard calibration methods fail if they do not account for censored observations.
Data-Dependent Tuning: Screening rules are typically tuned by searching over multiple thresholds to maximize yield (number of selected patients) while satisfying a risk constraint. Standard pointwise confidence intervals fail here because they do not account for the multiple testing problem introduced by this search, leading to invalid guarantees. Conversely, uniform confidence bands are often too conservative, resulting in low yield.

2. Methodology

The authors propose two complementary, distribution-free frameworks that utilize Inverse Probability of Censoring Weighting (IPCW) to handle right-censored data. Both frameworks assume access to a "black-box" survival model $\hat{S}$ and a censoring model $\hat{G}$ .

A. High-Probability (HP) Risk Control (Inductive Perspective)

This approach aims to guarantee that the event rate among selected patients is below a target $\alpha$ with high probability ( $1-\delta$ ). It extends the Learn–Then–Test (LTT) framework to censored data.

Mechanism: Instead of searching greedily over all thresholds (which inflates Type I error) or using overly conservative uniform bands, LTT performs fixed-sequence testing.
Procedure:
1. Define a set of candidate thresholds (paths) moving from conservative to liberal.
2. Compute pointwise upper confidence bounds (UCB) for the risk at each threshold using IPCW estimates.
3. Sequentially test thresholds along the path, stopping immediately before the first "unsafe" threshold (where UCB $> \alpha$ ).
4. The final threshold is the most liberal one accepted across the paths.
Guarantee: Provides a high-probability guarantee ( $P(\text{Risk} > \alpha) \le \delta$ ) that is less conservative than uniform bounds but more robust than naive pointwise selection.

B. False Discovery Rate (FDR) Control (Transductive Perspective)

This approach reformulates screening as a hypothesis testing problem on future patient outcomes, controlling the expected fraction of selected patients who experience the event.

Mechanism: Uses Selective Conformal P-values combined with the Benjamini–Hochberg (BH) procedure.
Procedure:
1. Construct "selective conformal p-values" for each patient in the target cohort. These p-values measure the evidence against the null hypothesis that the patient will experience the event before $t_0$ .
2. The p-values are constructed using IPCW to adjust for censoring in the calibration data.
3. Apply the BH procedure to these p-values to select the set of patients to reject (i.e., select as low-risk).
Guarantee: Controls the FDR (expected false discovery proportion) at level $\alpha$ . This provides an expectation-level guarantee rather than a high-probability one.
Innovation: The paper introduces a time-shift parameter ( $\gamma$ ) in the nonconformity scores to improve selection yield and a stabilization step to handle cases where no patients are selected.

3. Key Contributions

LTT for Censored Data: The first application of the Learn–Then–Test framework to right-censored time-to-event data, offering a practical balance between efficiency and safety guarantees.
Refined FDR Screening: A refined implementation of FDR-based screening for censored data using selective conformal p-values and IPCW, including a new tuning parameter ( $\gamma$ ) that significantly boosts yield while preserving theoretical guarantees.
Theoretical Unification: The paper establishes a theoretical link (Theorem 2) between FDR control and the selected-set event rate, showing they are effectively equivalent when the selection set is non-empty with high probability.
Empirical Comparison: A comprehensive comparison of HP-based methods (Pointwise, Uniform, LTT) and FDR-based methods against benchmarks (naive model-based and oracle) using semi-synthetic data and real oncology data.

4. Results

The methods were evaluated using:

Semi-synthetic data: Generated from the Flatiron Health Research Database (FHRD) with known ground truth to allow full evaluation.
Real Data: Direct application to FHRD data for advanced non-small cell lung cancer (aNSCLC) patients.

Key Findings:

Trade-off between Efficiency and Strength:
- FDR-Conformal: Consistently achieved the highest yield (selected the most patients) while maintaining the target survival rate on average. It is the most powerful method but offers an expectation-level guarantee.
- HP-LTT: Offered a strong middle ground. It provided high-probability guarantees (stronger than FDR) with yields significantly higher than uniform calibration and comparable to FDR in many settings.
- HP-Uniform: The most conservative method, often resulting in very low yields (few patients selected) to ensure strict safety.
- Naive Pointwise: Failed to control risk when data-dependent tuning was involved, often selecting too many high-risk patients.
Robustness: All calibrated methods successfully controlled the event rate below the target (e.g., 10%) even when the underlying survival model was misspecified (e.g., using Gradient Boosting on data generated by Random Forests).
Sample Size Sensitivity: As the calibration sample size decreased, the variability of all methods increased, and yields dropped, but the FDR and LTT methods remained robust in maintaining the target risk level.

5. Significance

This paper provides a critical toolkit for precision medicine and clinical trial design.

Safety vs. Opportunity: It solves the dilemma of balancing patient safety (avoiding high-risk patients in low-intensity trials) with scientific opportunity (not excluding eligible low-risk patients due to overly conservative rules).
Model Agnosticism: By treating predictive models as black boxes and relying on distribution-free calibration, these methods are applicable to any modern machine learning survival model.
Handling Censoring: The integration of IPCW into conformal inference and LTT frameworks allows these rigorous statistical guarantees to be applied to real-world clinical data, which is almost always right-censored.
Practical Guidance: The authors provide clear guidance on implementation, including the trade-offs between choosing a high-probability guarantee (LTT) versus an expectation-based guarantee (FDR), and how to tune parameters for optimal yield.

In summary, the paper demonstrates that FDR-based screening is typically the most powerful approach for maximizing patient selection, while LTT-based calibration offers a robust alternative for scenarios requiring strict high-probability safety guarantees, both significantly outperforming traditional calibration strategies in the presence of right-censoring.