Semiparametric KSD test: unifying score and distance-based approaches for goodness-of-fit testing

1. Problem Statement

The paper addresses the Goodness-of-Fit (GoF) testing problem, specifically the challenge of testing whether observed data $X_1, \dots, X_n$ follows a parametric model family $\{L_\theta : \theta \in \Theta\}$ against a general nonparametric alternative.

• The Core Challenge: Classical tests like the Likelihood Ratio Test (LRT) often fail or lose power when the true distribution lies outside the assumed parametric family. Conversely, nonparametric distance-based tests (e.g., Kolmogorov-Smirnov, Wasserstein distance) are powerful but suffer from the curse of dimensionality (computationally intractable in high dimensions) and often require smoothing techniques.
• The Gap: While score-based tests (like Rao's score test) are computationally efficient and avoid fitting the full alternative model, extending them to powerful nonparametric settings has been difficult due to the lack of suitable "scores" for nonparametric alternatives.

2. Methodology

A. Theoretical Unification: Score-Based Tests as IPMs

The authors establish a fundamental theoretical bridge between score-based tests and Integral Probability Metrics (IPMs).

• Exponentially Tilted Models (ETMs): They consider a class of models where the alternative is an exponential tilt of the null model: $L_{(\gamma, \theta)}(x) \propto p_\theta(x) \exp(\gamma f(x))$.
• Equivalence: They prove that the score function for the tilt parameter $\gamma$ in an ETM is equivalent to the difference in expectations of a test function $f$ under the empirical distribution and the null model.
• Implication: Maximizing this score over a rich function class $\mathcal{F}$ is mathematically equivalent to computing an IPM indexed by $\mathcal{F}$. This reinterprets classical distance-based tests (K-S, Wasserstein-1, MMD) as nonparametric score-based tests under specific ETMs.

B. The Proposed Solution: Semiparametric KSD (SKSD)

Building on this duality, the authors propose a new test called the Semiparametric Kernelized Stein Discrepancy (SKSD) test.

• Kernelized Stein Discrepancy (KSD): They utilize the KSD, an IPM defined over a Reproducing Kernel Hilbert Space (RKHS) induced by kernelized Stein functions.
• Semiparametric Nature: Unlike standard KSD which assumes a point-null (known $\theta$), the SKSD test handles composite null hypotheses where the parameter $\theta$ is unknown and must be estimated ($\hat{\theta}_n$) from the data.
• Test Statistic:
  $$T_n(X, \hat{\theta}_n) = \text{KSD}^2(L_{\hat{\theta}_n}, \hat{L}_n) = \frac{1}{n^2} \sum_{i,j=1}^n h_{\hat{\theta}_n}(X_i, X_j)$$
  where $h$ is a closed-form kernel function derived from the score function $\nabla_x \log p_{\hat{\theta}_n}(x)$ and the kernel $K$.
• Key Advantage: Due to Stein's Identity, the test statistic depends only on the score function and the kernel, not the likelihood. This makes it applicable to models with intractable likelihoods (e.g., exponential family graphical models, energy-based models) as long as the score is tractable.

C. Inference and Calibration

• Parametric Bootstrap: Since the asymptotic null distribution of $T_n$ is a complex infinite weighted sum of $\chi^2$ variables (dependent on the nuisance parameter estimator), the authors propose a parametric bootstrap procedure to calibrate the test.
• Agnostic Validity: They prove that this bootstrap procedure is valid for any nuisance parameter estimator $\hat{\theta}_n$ that satisfies a Uniform Asymptotic Linear Estimate (UALE) condition (covering M-estimators, minimum distance estimators, etc.).
• Stable Convergence: The theoretical proof relies on stable convergence theory to show that the bootstrap distribution converges to the true limiting distribution under both the null and alternative hypotheses.

3. Key Contributions

1. Unification Framework: The paper provides a novel theoretical perspective unifying score-based and distance-based GoF tests via Exponentially Tilted Models, showing that many classical IPM tests are essentially nonparametric score tests.
2. SKSD Test: Introduction of the SKSD test, which combines the computational efficiency of KSD with the flexibility of handling composite null hypotheses.
3. Computational Efficiency:
   • The test statistic has a closed-form solution computable in $O(n^2)$ time (or $O(n)$ with specific kernels).
   • This is significantly faster than K-S ($O(n^2)$ in 2D, NP-hard in high D) and Wasserstein-1 ($O(n^3)$) for multivariate data.
   • No numerical integration or Monte Carlo sampling is required for the statistic calculation.
4. Intractable Likelihoods: The method enables GoF testing for complex models (e.g., Kernel Exponential Families, Conditional Gaussian Graphical Models) where the normalizing constant is intractable, provided the score function is available.
5. Theoretical Guarantees:
   • Universal Consistency: The test is consistent against any fixed alternative distribution separated from the null in KSD.
   • Pitman Efficiency: The test attains optimal local power (Pitman efficiency) under local alternatives scaling at $n^{-1/2}$.
   • Bootstrap Consistency: Proven validity of the parametric bootstrap for power analysis under both null and alternative hypotheses.

4. Experimental Results

The authors validate the SKSD test through extensive simulations and real-world applications:

• Normality Testing: In testing for Gaussianity, SKSD achieves power comparable to specialized tests like Anderson-Darling and Lilliefors, despite being designed for general GoF problems.
• Kernel Exponential Family: The test successfully detects the correct rank (order) of the model, demonstrating its utility in model selection for nonparametric exponential families.
• Conditional Gaussian Graphical Models: Applied to models with intractable likelihoods, SKSD effectively detects the underlying graph structure (ring structure vs. higher-order interactions).
• Comparison: In high-dimensional settings, SKSD outperforms or matches the power of K-S, Wasserstein-1, and MMD tests while being computationally feasible. Notably, MMD tests often required numerical integration and showed lower power or higher computational cost in these settings.

5. Significance

This work is significant because it:

• Bridges a Theoretical Gap: It resolves the disconnect between the efficiency of score-based tests and the power of nonparametric distance-based tests.
• Enables New Applications: It provides a practical, scalable tool for diagnosing complex probabilistic models (common in modern machine learning) where likelihoods are intractable but scores are computable.
• Offers a Unified View: By showing that K-S, Wasserstein, and MMD are special cases of score tests under specific tilts, it offers a unified framework for understanding and designing GoF tests.
• Robustness: The reliance on general nuisance estimators (UALE) makes the method highly flexible for various statistical modeling contexts without requiring custom derivations for each specific estimator.

In summary, the SKSD test represents a major advancement in nonparametric GoF testing, offering a computationally efficient, theoretically rigorous, and broadly applicable solution for modern statistical and machine learning problems.