Sequential Control of False Positives in Online Change Point Detection

Technical Summary: Sequential Control of False Positives in Online Change Point Detection

Problem Statement
Online change point detection involves identifying distributional shifts in time-ordered data streams in real-time. This is particularly critical in mobile health (mHealth) applications, where continuous monitoring of behavioral or physiological data via smartphones and wearables can facilitate early intervention for psychiatric populations. However, performing sequential hypothesis tests as new data arrive creates a multiple testing problem. Traditional methods for controlling the Family-Wise Error Rate (FWER), such as Bonferroni or Šidák corrections, are ill-suited for this setting for two primary reasons:

1. High Dependence: Test statistics are computed using overlapping windows of data, inducing strong temporal correlation that standard corrections fail to account for, often leading to overly conservative thresholds.
2. Random Number of Tests: Unlike standard multiple testing with a fixed number of hypotheses, online monitoring continues until a change is detected, meaning the total number of tests is random and unbounded.

Consequently, existing sequential testing methods (e.g., group sequential designs or alpha-spending approaches) are difficult to apply directly because they typically assume a finite, pre-specified monitoring schedule. There is a need for a method that controls the probability of false alarms within a clinically meaningful, moving monitoring window while accounting for the inherent dependence of sequential tests.

Methodology
The authors propose a Sequential Family-Wise Error Rate (sFWER) and a corresponding simulation-based calibration procedure.

• Definition of sFWER: Unlike the classical FWER defined over a fixed set of tests, sFWER is defined as the probability of at least one false positive (Type I error) occurring within a moving monitoring window of length $\Delta$. The goal is to ensure that for any time $t$ after a run-in period $r$, the probability of observing at least one false alarm in the interval $[t+1, t+\Delta]$ is bounded by a target level $\alpha_\Delta$.
• Test Statistic: The authors utilize the VC* test statistic, a variance component score test for online multivariate change point detection. On each monitoring day $t$, the procedure evaluates candidate change points $k$ within a search window of $d_b$ days prior to $t$. For each candidate, a permutation-based p-value ($p_k$) is computed. The monitoring statistic $S_t$ is defined as the minimum of these p-values across the candidate days: $S_t = \min_{t-d_b \le k \le t-1} p_k$.
• Calibration Procedure: Because the joint distribution of the sequential statistics is non-trivial due to temporal dependence, an analytic cutoff is intractable. The authors propose a simulation-based approach:
  1. Generate $B$ datasets of length $r + \Delta$ under the global null hypothesis (no change point).
  2. For each dataset, compute the sequence of monitoring statistics and record the minimum statistic ($M$) over the monitoring window.
  3. Define the cutoff $c$ as the empirical $\alpha_\Delta$-quantile of the distribution of these minimum statistics.
  4. During actual monitoring, a change point is declared at day $t$ if $S_t \le c$.

Key Contributions

1. Formalization of sFWER: The paper introduces a new error metric tailored to online monitoring, shifting the focus from a fixed set of tests to a moving window of time, which better reflects clinical monitoring needs.
2. Simulation-Based Calibration: The authors develop a practical, resampling-based procedure to estimate monitoring thresholds that account for the complex dependence structure of sequential tests, avoiding the pitfalls of analytic approximations.
3. Empirical Validation: Through extensive simulation studies, the paper demonstrates that the proposed method achieves the desired error control, whereas standard unadjusted procedures lead to inflated false alarm rates, and traditional corrections (Bonferroni, Šidák) are overly conservative, reducing power.

Results

• Error Control: In simulation studies covering univariate and multivariate settings with varying run-in lengths ($r$) and window sizes ($\Delta$), the proposed procedure maintained rejection rates close to the target sFWER levels (0.1 and 0.2).
• Comparison with Alternatives:
  • Unadjusted: Produced significantly inflated false alarm rates, particularly as the monitoring window length increased.
  • Bonferroni/Šidák: Were overly conservative, resulting in rejection rates well below the target level and reduced power to detect true changes.
  • Proposed Method: Achieved a superior balance, maintaining high detection power for true change points (mean shifts of magnitude $\tau$) while strictly controlling false alarms.
• Robustness: The calibrated cutoff was found to be relatively insensitive to the length of the run-in period ($r$), allowing for flexibility in implementation.
• Real-World Application: The method was applied to passively collected smartphone GPS and mobility data from 41 adolescents and young adults. The procedure successfully identified 86 change points across 34 participants, demonstrating its feasibility in a real-world mHealth setting.

Significance and Claims
The paper claims that the proposed framework offers a more appropriate solution for controlling false positives in online monitoring settings than existing multiple testing corrections. By directly targeting the sFWER over a moving window and utilizing simulation to handle dependence, the method avoids the excessive conservatism of Bonferroni/Šidák corrections and the uncontrolled error rates of unadjusted testing.

The authors note that while motivated by mHealth, the framework is applicable to any setting involving repeated sequential monitoring. They modestly acknowledge limitations, including the computational intensity of the simulation step (though deemed negligible for daily monitoring) and the assumption of stable data under the null hypothesis. Furthermore, the real-world application was retrospective; the authors state that determining whether detected change points correspond to clinically meaningful events requires future prospective evaluation with clinical outcome data.