📊 statistics

Post-selection inference for quantifying uncertainty in changes in variance

This paper introduces two general approaches for constructing valid post-selection p-values to quantify uncertainty in detected changes in variance, addressing the bias and anti-conservative results that arise from naive double-use of data in change-point detection.

Original authors: Rachel Carrington, Paul Fearnhead

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rachel Carrington, Paul Fearnhead

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to find the exact moment a crime happened in a long video recording. You scan the footage, spot a suspicious moment, and then immediately turn around and ask, "How likely is it that this suspicious moment was just a random glitch?"

The problem, as this paper explains, is that if you use the same video to both find the suspicious moment and judge its likelihood, you are cheating. You've already seen the glitch, so your judgment will be biased. You'll think it's a real crime when it might just be noise. In statistics, this is called "using the data twice," and it leads to false alarms.

This paper introduces a new way to be a fair detective, specifically for finding moments when the volatility (or "jitteriness") of data changes, rather than just the average value.

Here is the breakdown of their solution using simple analogies:

1. The Problem: The "Double-Dip" Trap

Imagine you are looking for a sudden change in the temperature of a room. You scan the thermometer and see a spike. If you then use that same spike to calculate how "statistically significant" it is, you are essentially asking the question, "Is this spike real?" using the evidence that already convinced you it was there.

In the past, statisticians had a clever trick to fix this for changes in average temperature (the mean). They would say, "Okay, let's pretend we only look at the data if it looks like a spike." This creates a fair playing field. However, this trick didn't work for changes in volatility (how much the temperature swings up and down). This paper fills that gap.

2. The Solution: The "What-If" Game

The authors propose a method called Post-Selection Inference. Think of it as a "What-If" simulation game.

When you find a suspicious moment (a changepoint), instead of just calculating a probability based on the raw data, you ask:

"If I were to slightly wiggle the data around this moment, how often would my detective algorithm still pick this exact moment as the 'suspicious' one?"

The Wiggling: They mathematically stretch and shrink the data points before and after the suspected moment.
The Test: They run their detection algorithm on these "wiggled" versions.
The Result: If the algorithm always picks this moment no matter how you wiggle the data, it's a very strong signal. If the algorithm only picks it when the data is in a very specific, narrow configuration, it might just be a fluke.

3. Two Ways to Play the Game

The paper offers two different strategies for running this "What-If" game, depending on how the detective originally found the change.

Strategy A: The "Square It" Trick (CUSUM)

Sometimes, a change in volatility looks like a change in the average of the squared numbers.

The Analogy: Imagine you are watching a car's speed. If the car speeds up and slows down wildly (high variance), the squares of those speeds will have a higher average.
The Method: The authors take the data, square every number, and then use a standard, well-known method for finding average changes. Because this method is already well-understood, they can calculate the "fair probability" (p-value) using a neat mathematical formula, like solving a puzzle with a known solution.

Strategy B: The "Simulation" Trick (Likelihood Ratio)

Sometimes, the "Square It" trick isn't perfect, especially if the changes are complex. The authors also developed a method that works directly with the likelihood of the data.

The Analogy: This is like trying to find a needle in a haystack where the haystack keeps changing shape. You can't write a simple formula for the answer.
The Method: They use a computer to run thousands of mini-simulations.
1. They generate thousands of "fake" versions of the data.
2. They check: "In how many of these fake versions does our algorithm still find this specific changepoint?"
3. To make this fast, they use a Gaussian Process. Think of this as a smart "guessing machine" that learns the pattern of the results from a few samples and predicts the rest, so they don't have to run millions of simulations. It's like tasting a spoonful of soup to guess the flavor of the whole pot, rather than eating the whole pot.

4. Why This Matters

The paper proves that by using these "What-If" games, the resulting probabilities (p-values) are honest.

Before: If you used the old "double-dip" method, you might get a p-value of 0.01 (thinking it's a 99% sure thing) when it was actually just random noise.
Now: The new method ensures that if there is no real change, the p-values will be spread out evenly (like a fair lottery), so you don't get fooled by false alarms.

5. Real-World Test

The authors tested their methods on:

Fake Data: They created computer simulations where they knew exactly where the changes were. Their method correctly identified the real changes and ignored the fake ones.
Stock Market Data: They applied it to the S&P 500 stock market returns. They found moments where the market became much more volatile. Crucially, their method was able to distinguish between "real" volatility spikes and "fake" ones that looked suspicious but were just random noise.

Summary

This paper provides a new rulebook for detectives of data volatility. It stops them from cheating by using the same evidence to find a clue and judge it. Instead, it forces them to play a "What-If" game, simulating thousands of alternate realities to see if their clue holds up. This ensures that when they say, "This change is real," they can actually trust that statement.

Technical Summary: Post-selection Inference for Quantifying Uncertainty in Changes in Variance

Problem Statement

Quantifying uncertainty in detected changepoints is a critical challenge in time series analysis. A naive approach involves using the same dataset first to detect changepoints and then to test their significance. This "double use" of data biases the test statistics, leading to anti-conservative $p$ -values and inflated Type I error rates. While post-selection inference (PSI) methods have been developed to address this by conditioning on the selection event (i.e., the fact that a specific location was chosen for testing), existing literature has primarily focused on changes in the mean or piecewise constant polynomial models. There is a lack of established methods for quantifying evidence for detected changes in variance.

Methodology

The paper proposes two distinct approaches to construct valid post-selection $p$ -values for detecting changes in variance within a univariate Gaussian model with fixed mean and piecewise constant variance ( $X_t = \mu + \sigma_t \epsilon_t$ ).

1. The Test Statistic

For a detected changepoint $\hat{\tau}$ , the authors define a null hypothesis $H_0$ stating that variance is constant within a window $(\hat{\tau}-h, \hat{\tau}+h)$ . The test statistic $\phi$ is defined as the proportion of the sum of squares in the window accounted for by the data prior to $\hat{\tau}$ :
$\phi = \frac{\sum_{t=\hat{\tau}-h+1}^{\hat{\tau}} X_t^2}{\sum_{t=\hat{\tau}-h+1}^{\hat{\tau}+h} X_t^2}$
Under $H_0$ , $\phi$ follows a Beta distribution, $\text{Beta}(h/2, h/2)$ .

2. Post-Selection Conditioning

To obtain valid $p$ -values, the distribution of $\phi$ must be conditioned on the event that $\hat{\tau}$ was selected as a changepoint by the detection algorithm. This requires conditioning on the set $S = \{ \phi : \hat{\tau} \in M(X'(\phi)) \}$ , where $X'(\phi)$ represents a re-parameterized dataset where the sum of squares is preserved but the proportion $\phi$ is varied. The conditional distribution is a Beta distribution truncated to the set $S$ .

The paper details two methods for determining $S$ based on the changepoint detection algorithm used:

Approach A: CUSUM-based Detection (Analytical)
This method treats the change in variance as a change in the mean of the squared data ( $Y_t = X_t^2$ ). It utilizes the Cumulative Sum (CUSUM) statistic within a binary segmentation framework.
- Mechanism: The authors adapt existing PSI techniques for mean changes. By conditioning on the order of detection and the direction of changes, the selection event $S$ can be defined by a set of linear inequalities in $\phi$ .
- Calculation: Since the CUSUM statistic is linear in the data, and the data transformation $X'(\phi)$ is linear in $\phi$ , the inequalities defining $S$ become linear in $\phi$ . This allows $S$ to be calculated analytically as a union of intervals.
Approach B: Likelihood-Ratio (LR) based Detection (Monte Carlo)
This method directly maximizes the likelihood ratio statistic for the piecewise constant variance model, which generally offers higher detection accuracy than the CUSUM approach, especially for small changes.
- Mechanism: The likelihood ratio statistic is a non-linear function of $\phi$ . Consequently, the set $S$ cannot be defined by simple linear inequalities.
- Calculation: The authors propose a Monte Carlo approach using Gaussian Processes (GP) to approximate the set $S$ $S$ .
  1. Sample $\phi$ values and evaluate whether $\hat{\tau}$ is detected (i.e., whether $\phi \in S$ ) by re-running the changepoint algorithm on the transformed data $X'(\phi)$ .
  2. Model the indicator function $I(\phi \in S)$ as a Gaussian Process with an exponential kernel.
  3. Use the posterior mean of the GP to estimate the conditional density of $\phi$ given $\phi \in S$ .
  4. Calculate the $p$ -value either directly from this estimated density or via importance sampling.

3. Extensions

The paper discusses extensions to handle:

Different Null Hypotheses: Adapting the window size based on neighboring changepoints.
Power Optimization: Reducing the conditioning set by sampling nuisance parameters (related to the distribution of data orthogonal to $\phi$ ) to increase statistical power, though this adds computational complexity.

Results

The authors validate their methods using simulated data and a real-world financial dataset (S&P 500 log returns).

Simulation (CUSUM): QQ plots demonstrate that under the null hypothesis (no change), the calculated $p$ -values follow a uniform distribution $U(0,1)$ , confirming the validity of the post-selection inference. Under the alternative hypothesis, the tests show power to detect changes, which increases with the window size $h$ and the magnitude of the variance change.
Simulation (Likelihood-Ratio): The GP-based approach successfully produces uniform $p$ -values under $H_0$ even with a relatively small number of samples ( $N=50$ ). The study finds that using the GP posterior directly often yields results comparable to or better than using importance sampling, particularly for smaller sample sizes.
Comparison: The Likelihood-Ratio approach consistently outperforms the CUSUM approach in detection accuracy, particularly for smaller variance changes, though it requires the more computationally intensive Monte Carlo/GP estimation for $p$ -values.
Financial Application: Applied to S&P 500 data, the method successfully distinguishes between true and spurious changepoints. While algorithms like binary segmentation and PELT may detect multiple close changepoints, the post-selection $p$ -values correctly identify many of these as non-significant, effectively controlling the false discovery rate.

Key Contributions and Significance

The primary contribution of this work is filling the gap in post-selection inference literature by extending these methods to changes in variance.

General Framework: The paper provides a general framework applicable to a range of change-detection methods (CUSUM, Binary Segmentation, PELT) and various hypotheses.
Analytical and Computational Solutions: It offers an analytical solution for CUSUM-based detection (leveraging linearity) and a novel computational solution using Gaussian Processes for Likelihood-Ratio based detection (handling non-linearity).
Validity: The methods produce valid $p$ -values that are uniform under the null hypothesis, correcting the bias inherent in naive re-testing of detected changepoints.
Practical Utility: The application to financial data demonstrates that these methods are effective in real-world scenarios, helping to filter out false positives generated by aggressive changepoint detection algorithms.

The authors conclude that while their work assumes i.i.d. Gaussian data, the strategies developed—particularly the use of Gaussian processes to approximate selection events where explicit characterization is difficult—may be applicable to other post-selection inference problems. They note that future work could extend these methods to models with changing means or autocorrelation.