Inference conditional on selection: a review

This paper reviews selective inference techniques that provide valid conditional guarantees for statistical questions derived from data, such as identifying winners or clusters, by addressing the limitations of classical methods in modern scientific workflows and demonstrating their application through simulations and single-cell RNA sequencing data.

The Gist

The Big Problem: The "Double-Dipping" Trap

Imagine you are a detective trying to solve a crime. You have a room full of suspects (data).

1. The Old Way (Classical Statistics): You pick a suspect before you look at the evidence, based on a hunch, and then you run a test to see if they are guilty. This is fair.
2. The Modern Problem (Double Dipping): In modern science, we often look at the whole room of suspects first, find the one who looks the most suspicious (the "winner"), and then run a test to see if they are guilty.

The Catch: If you pick the most suspicious person just because they look the most suspicious, you are almost guaranteed to be wrong about how "guilty" they actually are. You've used the same evidence to pick the suspect and to judge them. This is called double dipping.

In statistics, this leads to the "Winner's Curse." If you pick the candidate with the highest test score, that score is likely inflated by luck. If you then calculate a confidence interval (a range of where the true score likely is) using standard math, that range will be too narrow. You will be overconfident, and your "95% sure" claim will actually only be right 50% of the time.

The Solution: Conditional Inference

The paper argues that we need a new way of thinking. Instead of asking, "Is this person guilty?" we should ask, "Given that we picked this specific person as the most suspicious, are they actually guilty?"

This is called Conditional Inference. It's like saying, "Okay, we already looked at the whole room and picked the guy with the messy hair. Now, let's re-evaluate his guilt only considering the fact that we picked him for having messy hair."

The paper reviews several "recipes" to fix this double-dipping problem without throwing away the data.

---

The Four Recipes for Fixing Double Dipping

The authors compare four main ways to solve this. Think of them as different ways to manage a team of investigators.

1. Full Conditional Selective Inference (The "Strict Judge")

• How it works: You use all the data to pick the suspect. Then, you act like a strict judge who says, "I know you picked him because he had the messiest hair. I will now calculate his guilt only looking at the specific scenario where he had the messiest hair."
• The Good: You use every single piece of evidence. You don't throw anything away.
• The Bad: It's incredibly hard to do the math. Sometimes, if the "messy hair" wasn't that much messier than the others, the math gets so complicated that the answer becomes "We have no idea" (an infinitely wide confidence interval). It's like a judge saying, "Because the evidence is so ambiguous, I can't give you a verdict."

2. Sample Splitting (The "Two-Team Approach")

• How it works: You split your team of investigators into two groups: Team A and Team B.
  • Team A looks at the suspects and picks the winner.
  • Team B (who has never seen the suspects before) tests the winner.
• The Good: It's easy. Team B has no bias because they didn't help pick the suspect.
• The Bad: You threw away half your data. Team A's findings are discarded after the selection. If Team B doesn't have enough information, they might give up and say, "I can't tell."

3. Data Thinning (The "Magic Filter")

• How it works: Instead of cutting the team in half, you use a "magic filter" on the data. You take the original data and split it into two independent streams of information.
  • Stream A is used to pick the winner.
  • Stream B is used to test the winner.
  • Crucially, Stream B still contains some information about the winner, even though it's independent of Stream A.
• The Good: You don't throw away data like in Sample Splitting. You get a verdict even when the data is tricky.
• The Bad: It only works if the data follows specific mathematical rules (like a bell curve). If your data is weird, this filter breaks.

4. Randomized CSI (The "Controlled Chaos")

• How it works: This is a mix of the above. You use the whole data to pick the winner, but you add a little bit of "noise" (random static) to the selection process.
  • Imagine adding static to a radio signal. You pick the station based on the noisy signal.
  • Then, you use the original clean signal to test the station, but you account for the fact that you picked it based on the noisy version.
• The Good: It prevents the "infinite verdict" problem of the Strict Judge. It uses all the data but keeps the math manageable.
• The Bad: You have to introduce artificial randomness, which can feel weird to scientists who want pure data.

---

Real-World Examples from the Paper

The authors tested these recipes on three real scenarios:

1. The "Winner's Curse" (Example 1): Picking the best-performing drug from a list of 100.
   • Lesson: If you pick the winner and test it normally, you overestimate its success. You need to adjust for the fact that you picked the "champion."
2. Regression Trees (Example 2): Using an algorithm to find subgroups of patients who respond well to a treatment.
   • Lesson: The algorithm finds the groups because of the data. If you then test those groups, you are double-dipping. The "Two-Team" (Sample Splitting) or "Magic Filter" (Data Thinning) approaches worked well here.
3. Single-Cell RNA Sequencing (Example 3): Grouping cells into types (like "T-cells" vs. "B-cells") and then checking which genes are different.
   • Lesson: This is the hardest case. You can't split the cells in half easily because if you cluster half, you don't know how to label the other half.
   • Result: The "Magic Filter" (Data Thinning) and "Controlled Chaos" (Randomized CSI) worked best. The "Strict Judge" (Full CSI) was too rigid and couldn't handle the messy biological data.

The Bottom Line

Science has moved from "hypothesis-driven" (guessing first, then testing) to "data-driven" (exploring first, then testing). The old math doesn't work for this new way of doing science because it leads to false confidence.

The paper concludes that there is no single "perfect" tool.

• If you want to use all your data and have a complex model, you might need the Strict Judge (Full CSI), but be prepared for wide, uncertain answers.
• If you want simplicity and don't mind throwing away some data, Sample Splitting is great.
• If you have clean, standard data, Data Thinning is a sweet spot.
• If you want a balance of using all data and getting a definite answer, Randomized CSI is often the winner.

The Takeaway: Scientists need to stop "double dipping." They must choose a method that acknowledges they picked their question from the data, not before it. The paper provides a menu of options to help them do that without losing their minds over the math.

Technical Summary

1. Problem Statement

Modern scientific workflows often involve data-driven selection, where models, hypotheses, or parameters are chosen based on exploratory analysis of the data itself (e.g., selecting the "winner" of a treatment trial, identifying regions in a regression tree, or clustering cells in single-cell RNA sequencing).

• The Core Issue: When statistical inference (e.g., confidence intervals, p-values) is performed on these data-selected parameters using classical methods (like standard t-tests or Wald intervals) without accounting for the selection process, the results suffer from "double dipping."
• Consequence: Classical methods fail to provide valid guarantees. Specifically, they do not achieve nominal coverage for confidence intervals or control the Type I error rate for hypothesis tests. This phenomenon is often referred to as the Winner's Curse (in the context of selecting the maximum) or the Replication Crisis.
• Goal: The paper reviews Selective Inference, a framework designed to provide valid statistical guarantees for parameters selected based on the data.

2. Methodological Framework: Conditional vs. Unconditional Coverage

The authors distinguish between two types of inferential guarantees:

• Unconditional Coverage: The probability that the interval covers the parameter, averaged over all possible selection events. While achievable via simultaneous inference (e.g., Bonferroni correction), the authors argue this is often scientifically uninteresting because it can lead to over-coverage in "correct" selections and under-coverage in "incorrect" ones.
• Conditional Coverage (Selective Coverage): The probability that the interval covers the parameter, conditional on the specific selection event having occurred (i.e., $P(\theta \in CI \mid S(Y)=k) \geq 1-\alpha$).
  • Argument: The authors advocate for conditional coverage as the primary goal. It ensures that inference is valid specifically for the parameter the scientist actually chose to investigate, preventing the "double dip" where information used for selection is reused for inference.

3. The "Unifying Recipe"

The paper proposes that nearly all valid selective inference methods follow a general three-step recipe:

1. Split Data: Divide the data (or information) into a Selection Set ($Y_{sel}$) and an Inference Set ($Y_{inf}$). These sets may be disjoint, overlapping, or even identical.
2. Select: Apply the selection algorithm $S(\cdot)$ to $Y_{sel}$ to identify the target parameter $\theta_{S(Y_{sel})}$.
3. Infer: Construct a confidence interval or test statistic on $Y_{inf}$, conditional on the event that the selection occurred (i.e., conditioning on $S(Y_{sel}) = k$ or a superset of this event).

4. Key Approaches Reviewed

The authors categorize existing methods based on how they implement the "Unifying Recipe" and manage the trade-off between selection quality and inference power (Fisher information).

A. Full Conditional Selective Inference (Full CSI)

• Mechanism: Uses the entire dataset for both selection and inference ($Y_{sel} = Y_{inf} = Y$).
• Conditioning: Conditions on the exact selection event (or a sufficient statistic implying it).
• Pros: Uses all available data; no information is discarded.
• Cons:
  • Requires deriving the exact conditional distribution of the test statistic given the selection event (often analytically intractable).
  • Fisher Information Trade-off: If the selection event is "borderline" (e.g., the winner is only slightly better than the runner-up), the conditional distribution becomes highly constrained, leading to infinitely wide confidence intervals or zero power.

B. Sample Splitting

• Mechanism: Splits data into disjoint sets ($Y_{sel} \cap Y_{inf} = \emptyset$). Selection happens on $Y_{sel}$; inference on $Y_{inf}$.
• Pros: Simple to implement; allows use of standard "off-the-shelf" inference tools; avoids infinite intervals.
• Cons: Discards information in $Y_{sel}$ that wasn't strictly necessary for selection. In non-i.i.d. settings (e.g., fixed covariates in regression trees), it can result in regions with no data in the inference set, leading to infinite intervals.

C. Data Carving

• Mechanism: A hybrid where $Y_{sel} \subset Y$ and $Y_{inf} = Y$.
• Pros: Uses all data for inference, avoiding the information waste of sample splitting.
• Cons: Requires complex conditional distributions (similar to Full CSI) because $Y_{sel}$ and $Y_{inf}$ are not independent.

D. Data Thinning

• Mechanism: Decomposes a random variable $Y$ into two independent components ($Y_{sel}, Y_{inf}$) via a transformation (e.g., $Y_{sel} = \epsilon Y + \zeta$).
• Pros: Maintains independence between selection and inference sets (allowing standard tools) while retaining information in both sets. Applicable to specific distributions (Gaussian, Poisson, etc.).
• Cons: Limited to specific distributional families; requires known nuisance parameters (e.g., variance).

E. Randomized Conditional Selective Inference (Randomized CSI)

• Mechanism: Injects noise into the selection step (e.g., randomized CART) or uses a randomized selection set, then conditions on the randomized selection.
• Pros: Avoids the "infinite width" problem of Full CSI by placing a lower bound on the information available for inference. More flexible than data thinning.
• Cons: Requires deriving a new conditional distribution for each specific selection rule.

F. Data Fission

• Mechanism: Decomposes $Y$ into $Y_{sel}$ and $Y_{inf}$ that are not independent but have tractable conditional distributions.
• Pros: Can be applied where data thinning fails (e.g., Bernoulli data, overdispersed negative binomial data).
• Cons: Inference requires handling the dependence between sets, which can be computationally challenging.

5. Simulation and Real-World Results

Simulation Study (Regression Trees)

The authors simulated inference on the mean of regions identified by a CART regression tree.

• Findings:
  • Classical methods failed to achieve 90% coverage (often <40% in weak signal settings).
  • Full CSI achieved perfect coverage but produced extremely wide intervals in weak signal scenarios.
  • Sample Splitting and Data Thinning produced finite intervals but suffered from reduced selection quality (poorer tree structure) when less data was allocated to selection.
  • Randomized CSI emerged as a strong compromise: it achieved valid coverage with significantly narrower intervals than Full CSI and better selection quality than Sample Splitting/Data Thinning, adapting well to signal strength.

Application: Single-Cell RNA Sequencing (scRNA-seq)

The authors applied these methods to clustering cells and testing for differential gene expression.

• Challenge: Sample splitting is impossible here because clustering one subset of cells does not provide labels for the other subset.
• Methods Tested: Classical, Poisson/Negative Binomial Thinning, Data Fission, and Full CSI.
• Results:
  • Classical methods produced highly anti-conservative p-values (false discoveries).
  • Selective methods (Thinning, Fission, Full CSI) successfully controlled the Type I error rate (p-values were closer to uniform in negative controls).
  • Trade-offs: Full CSI was sensitive to preprocessing and clustering algorithms (limited to k-means/hierarchical). Data Thinning and Fission introduced randomness in the clustering step (scientists might get different clusters on the same data), but offered more flexibility with distributional assumptions.

6. Significance and Contributions

1. Conceptual Unification: The paper provides a cohesive framework ("The Recipe") that unifies diverse selective inference methods, clarifying their relationships and the fundamental trade-off between selection quality (using more info for selection) and inference power (leaving info for inference).
2. Advocacy for Conditional Guarantees: It strongly argues that conditional coverage is the scientifically relevant standard for data-driven questions, as it protects against the specific biases introduced by the selection process.
3. Practical Guidance: Through simulations and real data, the authors demonstrate that no single method is universally superior.
   • Full CSI is powerful but rigid and prone to infinite intervals.
   • Sample Splitting is simple but wasteful.
   • Randomized CSI and Data Fission offer promising middle grounds, balancing flexibility and power.
4. Future Directions: The authors highlight the need for assumption-lean methods and general-purpose software to make selective inference accessible to non-statisticians, particularly in complex fields like genomics and neuroscience.

In summary, this review serves as a comprehensive guide for researchers navigating the pitfalls of data-driven inference, offering a taxonomy of solutions and practical insights into how to balance the competing demands of exploration and rigorous statistical validation.