📊 statistics

A Two-Stage Detection Framework for Interaction Screening and Testing: A Systematic Simulation Evaluation of Method Suitability

This study proposes and systematically evaluates a two-stage "screen-plus-confirm" framework that combines SHAP-based top-k screening with Wald tests, BH correction, and bootstrap robustness validation to effectively balance false-positive control and detection sensitivity in interaction analysis, while establishing specific operational guidelines and evidence-grading levels for real-world application.

Original authors: Yue-peng Liu

Published 2026-10-27

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yue-peng Liu

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast landscape of medical research, scientists often look for the main drivers of health and disease, such as how a specific gene or a lifestyle factor influences an outcome. However, the story is rarely that simple. Often, the true cause lies in the relationship between two factors working together, a phenomenon known as an interaction. For instance, a drug might work wonders for one group of patients but do nothing for another, depending on their age or another biological marker. Finding these hidden partnerships is crucial for understanding how the body truly functions, yet it is a notoriously difficult task. The challenge arises because the number of possible pairs to check grows explosively as more variables are added; with just a few dozen factors, researchers face thousands of potential combinations, most of which are just random noise. To make matters worse, the tools used to find these connections often struggle to tell the difference between a genuine biological partnership and a false signal created by the way the data is structured.

To solve this puzzle, a researcher named Yue-peng Liu developed a new, two-step approach to sift through the noise and find the real interactions. The study, which relies heavily on computer simulations to test its ideas against known truths, proposes that no single method can do the whole job alone. Instead, the process must be split into two distinct phases: a broad screening to narrow down the list of suspects, followed by a rigorous confirmation to prove which ones are real. The researcher tested this idea by creating thousands of fake datasets where the answers were already known. In some of these simulations, there were no interactions at all, just to see if the methods would falsely claim to find one. In others, there were clear interactions, including some that were simple and linear, and others that were complex and only appeared when a variable crossed a specific threshold. By running these scenarios over and over, the study could measure exactly how often each method succeeded or failed.

The results revealed a clear weakness in relying on a single tool. Methods based on machine learning, which are excellent at spotting complex patterns, turned out to be too eager. When the data contained no real interaction, these tools frequently pointed to a false partnership, claiming to find a connection where none existed. On the other hand, traditional statistical tests, which are very strict about proving a connection, often missed the more complex, non-linear interactions entirely. The solution found was to use the strengths of both. The new framework starts by using a machine-learning tool to quickly scan all possible pairs and pick the top five most promising candidates. This step is like casting a wide net to ensure the real signal isn't missed. However, because this net catches a lot of false alarms, the next step is critical. The five candidates are then subjected to a strict statistical test that checks for significance, followed by a correction for the fact that multiple guesses were made. Finally, to ensure the result isn't just a fluke of the specific sample, the data is resampled many times to see if the finding holds up consistently.

This two-stage process proved to be highly effective in the simulations. By combining the broad reach of the screening with the strict discipline of the confirmation, the framework reduced the rate of false alarms significantly, bringing the false confirmation rate down to 7% in the final configuration, while still successfully identifying 83 percent of the real, complex interactions. The study also tested how robust this method was under different conditions, such as when the sample size was small or when the variables were highly correlated. It found that the method requires a sample size of at least 500 people to work reliably and performs best when the event being studied, like a disease outcome, happens in at least 30 percent of the group. Interestingly, the method worked even better for survival data, where the timing of an event matters, than for simple yes-or-no outcomes. The research also explored whether removing the correlation between variables before starting would help, but found that this actually destroyed the signal of real interactions, suggesting that the strict confirmation stage is the better way to handle these issues.

To see if this framework worked in the real world, the researcher applied it to two existing medical datasets. In one study involving patients with acute kidney injury, the method scanned nearly 600 possible pairs of variables and narrowed them down to a single candidate that showed a statistically supported interaction between phosphate levels and a comorbidity index. This finding was not just a simple "yes" or "no"; the framework allowed for a deeper look, revealing that the effect of phosphate on mortality risk was stronger in patients with fewer other health problems. In a second dataset involving genetic data and stroke risk, the framework was used to re-examine a previously published claim of a significant gene-gene interaction. The new analysis rejected this claim, showing that the original finding was likely a result of a weak signal and a less rigorous testing method. This case was essentially a negative result where the framework successfully rejected a reported interaction, rather than a positive confirmation of detection capability, highlighting the method's ability to identify when a claimed interaction is not reproducible under stricter standards. This ability to both find new leads and confidently discard false ones demonstrates the power of the approach.

The study concludes that the best way to find interactions is not to rely on a single, all-powerful method, but to use a pipeline that respects the different needs of discovery and proof. It recommends a specific workflow: screen with a machine-learning tool to find the top five candidates, test them with a standard statistical method, correct for multiple guesses, and then verify the result by checking if it repeats consistently in resampled data. This process creates a four-level system of evidence, ranging from a simple candidate indication to a robust confirmation, ensuring that researchers do not mistake a lucky guess for a scientific fact. While the method is not perfect and still faces challenges when dealing with very strong correlations between variables, it provides a clear, tested path forward for medical and epidemiological research, turning a chaotic search through thousands of possibilities into a manageable and reliable process.

Technical Summary: A Two-Stage Detection Framework for Interaction Screening and Testing

Problem Statement

Interaction analysis in clinical and epidemiological research faces a fundamental tension between two progressive demands: screening and testing.

Screening: With $p$ predictors, there are $p(p-1)/2$ potential interaction pairs. The candidate space grows quadratically, and the vast majority are noise or spurious interactions. Efficient methods are required to rapidly shrink this space to a small set worthy of examination without incurring prohibitive computational costs.
Testing: Confirming whether a candidate interaction reflects a real effect requires methods with strict significance control and uncertainty quantification.

Existing methods fall into distinct paradigms with complementary but conflicting strengths:

Screening-oriented methods (e.g., SHAP interaction values, Friedman H statistic, Sure Independence Screening extensions) offer high coverage and computational efficiency but often lack significance control, leading to high false-positive rates, particularly under collinearity.
Testing-oriented methods (e.g., product-term Wald tests, GAM tensor-product terms) provide strict significance control but often suffer from low power when interactions are nonlinear (e.g., threshold-type effects) or when the functional form is misspecified.

Current literature lacks systematic evidence on which methods suit the screening stage versus the testing stage, or how to construct an operational "screen-plus-confirm" pipeline that balances coverage with false-positive control.

Methodology

The study adopts a framework of "AI-assisted screening + simulation with known truth + public-data validation" to evaluate method suitability and construct a two-stage detection pipeline.

1. Simulation Framework

Three scenarios were generated with $n=1000$ samples and $p=10$ predictors (including collinear variables):

Scenario A (No Interaction, Collinear): Contains only main effects with collinear predictors ( $X_1, X_2$ ). Used to evaluate false-positive rates.
Scenario B (Linear Multiplicative Interaction): Adds a linear interaction term ( $0.8 \cdot X_1 \cdot X_2$ ). Used to evaluate linear interaction detection.
Scenario C (Threshold-Type Nonlinear Interaction): Adds a threshold interaction ( $0.8 \cdot X_2 \cdot I(X_1 > 0.5)$ ). Used to evaluate detection of nonlinear forms that linear product terms cannot express directly.

2. Comparator Methods

Five representative methods were compared:

Product-term Wald test: Parametric testing.
GAM tensor-product interaction: Semiparametric smoothing.
SHAP interaction top-1: Machine-learning attribution.
Friedman H statistic: Nonparametric interaction strength.
Model-based recursive partitioning (MOB): Structural change detection.

3. The Two-Stage Detection Pipeline

The proposed framework consists of:

Screening Stage: An XGBoost model is trained to compute SHAP interaction strengths ( $S_{ij}$ ) for all pairs. The top- $k$ pairs form the candidate set $C$ .
Confirmation Stage:
- A logistic (or Cox) model is fitted for each candidate pair including all main effects and the product term.
- The Wald test $p$ -value is calculated and corrected using the Benjamini–Hochberg (BH) procedure.
- Pairs passing BH correction undergo $B$ bootstrap resamples to compute a reproducibility metric $\rho_{ij}$ (proportion of significant resamples).
- Pairs with $\rho_{ij} \geq \tau$ (threshold) are robustly confirmed.

4. Refinement and Validation

The study systematically refined engineering parameters through comparative experiments:

Screening Width ( $k$ ): Evaluated $k \in \{1, 3, 5, 10, 15\}$ .
Screener Family: Compared SHAP (model attribution) vs. SIS-type (statistical test ranking).
Robustness Grid: Tested performance across collinearity ( $\rho$ ), sample size ( $n$ ), event rates, and outcome types (binary vs. Cox survival).
Preprocessing: Evaluated collinearity residualization and permutation noise calibration.
Mechanism Interpretation: Validated MOB as an independent evidence channel vs. a post-hoc refinement tool.
Real-Data Case Studies: Applied the framework to the CRRT clinical cohort ( $n=1144$ ) and GMDR public data ( $n=2377$ ).

Key Contributions

Quantification of Failure Modes: The study demonstrates that single methods fail in specific regimes: SHAP has a 64% false-positive rate under collinearity without interaction, while linear product-term tests detect only 85% of threshold-type nonlinear interactions.
Operational Two-Stage Pipeline: The study constructs and validates a specific pipeline: SHAP interaction top- $k$ screening ( $k=5$ ) $\rightarrow$ Product-term Wald test $\rightarrow$ BH correction $\rightarrow$ Bootstrap reproducibility ( $\geq 0.9$ ).
Evidence-Grading Framework: A four-level evidence system is established:
- Level 1: Candidate indication (SHAP top- $k$ ).
- Level 2: Preliminary evidence (Wald $p < 0.05$ , no BH).
- Level 3: Preliminary confirmation (BH passed, bootstrap $< 0.9$ ).
- Level 4: Robust confirmation (BH passed, bootstrap $\geq 0.9$ ).
Method Suitability Conclusions:
- Screening: Tree-model attribution (SHAP) is superior to statistical-test ranking (SIS) because it provides informationally heterogeneous signals, reducing the "false-confirmation" rate in the confirmation stage.
- Confirmation: Parametric/semiparametric tests are essential for significance control.
- MOB Role: MOB has high sensitivity but uncontrolled false positives (26% in no-interaction scenarios). It is not suitable as independent confirmation evidence but serves effectively as a mechanism-refinement tool for confirmed interactions.
Falsification of Alternative Paths: The study empirically rules out collinearity residualization (which removes true interaction signals) and permutation noise calibration (which offers limited benefit at high computational cost) as effective preprocessing strategies.

Results

Performance: The two-stage method reduced the end-to-end false-positive rate in the no-interaction scenario from 64% (SHAP alone) to ~5% (with $k=1$ ) and further to near zero with BH and bootstrap confirmation, while maintaining 83% detection in the nonlinear scenario.
Optimal Parameters:
- Screening Width: $k=5$ offers the best compromise (99% coverage of true interactions in Scenario C).
- Bootstrap Threshold: Raising the threshold to 0.9 reduces false confirmations significantly (e.g., from 15% to 7% in Scenario A) at the cost of slightly lower detection in nonlinear scenarios (47%).
- Sample Size: A clear threshold exists; detection collapses at $n < 500$ and becomes reliable only at $n \geq 500$ .
- Outcome Type: The framework performs robustly on both binary and survival (Cox) outcomes, with unexpectedly higher detection rates in survival data.
Real-Data Applications:
- CRRT Cohort: Screened 595 pairs down to one candidate ( $P24h \times CCI$ ), assigned Level 3 "preliminary confirmation." MOB refinement revealed a unidirectional mechanism where comorbidity burden modifies the phosphate effect.
- GMDR Data: Rejected a literature-reported gene-gene interaction ( $ITGA2 \times HABP2$ ) that failed to pass the top-5 screening and showed no statistical significance under global testing, demonstrating the framework's ability to reject spurious interactions.

Significance and Claims

The paper claims that the "screen–confirm" division of labor is necessary because no single method can simultaneously satisfy the demands of high coverage and strict significance control. The proposed framework provides:

A Methodological Standard: A simulation-evidence-based reference for method suitability in medical and epidemiological settings.
Operational Boundaries: Clear usage conditions ( $n \geq 500$ , event rate $\geq 30\%$ , collinearity $\rho \leq 0.7$ ) and a structured evidence-grading system to prevent overinterpretation of screening scores.
Dual Capability: The ability to both "screen out candidates" (converging from hundreds to a few) and "reject spurious interactions" (validating against literature claims), thereby addressing the false-positive propagation often seen in interaction studies.

The study concludes that while the framework balances the trade-off between false positives and sensitivity, it acknowledges inherent limitations, particularly the difficulty of distinguishing collinear spurious interactions from true linear interactions in strong collinearity settings, which may require multiple evidence sources or domain priors.