Perturbed Double Machine Learning: Nonstandard Inference Beyond the Parametric Length

1. Problem Statement

The paper addresses the challenge of constructing valid confidence intervals for low-dimensional functionals (e.g., treatment effects) in the presence of infinite-dimensional nuisance parameters estimated via machine learning (ML).

• The Limitation of Standard DML: Classical Double/Debiased Machine Learning (DML) relies on the assumption that nuisance estimators (e.g., $f(X) = E[D|X]$ and $g(X) = E[Y|X]$) converge at a rate faster than $n^{-1/4}$. Under this condition, the nuisance bias term becomes asymptotically negligible ($o_P(n^{-1/2})$), allowing the DML estimator to be asymptotically Gaussian and enabling the construction of valid Wald intervals.
• The Failure Regime: In many modern applications involving complex, high-dimensional, or non-smooth data (e.g., dense models, non-sparse structures, or complex black-box ML), nuisance estimators may converge slower than $n^{-1/4}$ or even $n^{-1/2}$. In these "non-regular" regimes, the nuisance bias dominates the estimation error, causing standard Wald intervals to suffer from severe undercoverage.
• The Gap: Existing methods for non-regular inference (e.g., higher-order influence functions) often require strong structural assumptions (like specific smoothness or sparsity) or are computationally intractable. There is a lack of "agnostic" methods that provide valid inference regardless of the nuisance estimator's convergence rate or the underlying data structure.

2. Methodology: Perturbed DML

The authors propose Perturbed Double Machine Learning (Perturbed DML), a two-step procedure that injects randomness into the nuisance estimation process to generate a collection of candidate estimators, followed by a filtering mechanism to construct a robust confidence set.

Step 1: Perturbation

Instead of relying on a single nuisance estimate, the method generates $M$ perturbed nuisance models.

• Mechanism: In the training phase (using a sample $I_c$), artificial noise is injected into the response variables ($Y$ and $D$) before fitting the nuisance models.
  • For linear models (Lasso), the noise is injected into the score vectors (gradients) of the optimization problem.
  • For general ML, noise is added directly to the response variables: $Y^{[m]}_i = Y_i - \epsilon^{[m]}_i$ and $D^{[m]}_i = D_i - \delta^{[m]}_i$.
• Noise Distribution: The noise is drawn from a distribution (typically Gaussian) with a covariance matrix estimated from the data. The key requirement is that the distribution is sufficiently "diffuse" so that, with high probability, at least one perturbation nearly cancels out the true noise in the data.
• Result: This yields a collection of $M$ perturbed nuisance estimators ($\hat{f}^{[m]}, \hat{g}^{[m]}$) and corresponding DML estimators ($\hat{\beta}^{[m]}$).

Step 2: Filtering and Aggregation

Simply taking the union of all $M$ Wald intervals would be overly conservative and potentially too wide. The authors introduce a filtering step:

• Filtering Criterion: The method discards perturbed intervals whose center $\hat{\beta}^{[m]}$ deviates significantly from the original unperturbed DML estimate $\hat{\beta}$.
• Threshold: The retention set $\mathcal{M}$ is defined as:
  $$\mathcal{M} = \{ m : |\hat{\beta}^{[m]} - \hat{\beta}| \leq \text{Threshold} \}$$
  The threshold is derived theoretically (based on the nuisance bias bound $\rho_n$) or practically (using an empirical quantile, e.g., the 95th percentile of deviations).
• Final Confidence Set: The final confidence interval is the union of the Wald intervals corresponding to the retained perturbations:
  $$CI = \bigcup_{m \in \mathcal{M}} CI^{[m]}$$

3. Key Contributions

1. Valid Inference Beyond $n^{-1/4}$: The paper proves that Perturbed DML achieves nominal coverage even when nuisance estimators converge slower than $n^{-1/4}$ (and potentially slower than $n^{-1/2}$), a regime where standard DML fails.
2. Oracle-Like Performance: Theoretically, the authors show that with a sufficiently large $M$, there exists at least one perturbation $m^*$ such that the perturbed nuisance estimates are sufficiently close to the true nuisance functions. Consequently, $\hat{\beta}^{[m^*]}$ behaves like an "oracle" estimator (one that knows the true nuisances), possessing the correct asymptotic distribution.
3. Optimality and Length Control:
   • The method achieves the minimax expected length for confidence intervals in high-dimensional sparse linear models (matching the lower bounds established by Cai and Guo, 2017).
   • The filtering step is crucial: it prevents the union of intervals from becoming excessively wide while ensuring the "oracle-like" interval is retained with high probability.
4. Agnosticism to Nuisance Learners: The framework extends to general machine learning algorithms (e.g., Random Forests, XGBoost, Neural Networks) without requiring specific structural assumptions (like sparsity or smoothness) on the nuisance functions, provided the perturbation noise is sufficiently dispersed.

4. Theoretical Results

• Theorem 1 (High-Dimensional Linear Models): Under Lasso estimation, if $M$ is large enough, there exists an $m^*$ such that $|\hat{\beta}^{[m^*]} - \hat{\beta}_{ora}|$ is small with high probability. The convergence rate depends on $M$ and the dimension $p$.
• Theorem 2 (Coverage and Length): The filtered union interval $CI$ satisfies:
  • Coverage: $\liminf P(\beta \in CI) \geq 1 - \alpha$.
  • Length: The length is of the order $O(\rho_n + n^{-1/2})$, where $\rho_n$ is the nuisance bias bound. This matches the minimax optimal rate.
• Theorem 3 (General ML): Using isoperimetric inequalities for Gaussian measures, the authors provide a justification for the general ML setting, showing that the perturbation step ensures the conditional distribution of $\hat{\beta}^{[m]}$ covers the true parameter's uncertainty region.

5. Empirical Results

• Simulation Studies:
  • Sparse Linear Models: In settings where the sparsity level $s$ is high (dense models), standard DML Wald intervals drop to ~70% coverage (target 95%). Perturbed DML maintains >95% coverage.
  • General ML: The method was tested with Generalized Additive Models (GAM) and XGBoost. In scenarios where standard DML failed due to slow convergence or bias, Perturbed DML delivered valid coverage.
  • Interval Length: The proposed intervals are only moderately wider than the "Oracle Bias-Aware" benchmark (which assumes knowledge of the true bias) and significantly shorter than naive bias-bound intervals (which add the worst-case bias bound directly to the interval).
• Real Data Analysis:
  • Applied to the effect of gun ownership on homicide rates (Chernozhukov et al., 2024 dataset).
  • Compared across various nuisance learners (OLS, Lasso, Random Forests, XGBoost, DNNs).
  • Finding: Perturbed DML produced wider intervals than standard DML (reflecting the uncertainty in slow-converging nuisances) but maintained robustness. Notably, for Deep Neural Networks (DNNs), which are sensitive to perturbations, the method widened the intervals significantly, correctly capturing the instability that standard DML missed.

6. Significance

This work represents a significant advancement in semiparametric inference and causal inference with machine learning.

• Robustness: It offers a "safety net" for practitioners using complex black-box ML models where the convergence rates of nuisance estimators are unknown or likely slow.
• Practicality: Unlike higher-order influence function methods, which are computationally heavy and require tuning complex basis dictionaries, Perturbed DML is relatively simple to implement (requiring only repeated model fitting with noise injection).
• Theoretical Breakthrough: It bridges the gap between the "regular" regime (where DML works) and the "non-regular" regime, providing a unified framework that does not rely on the estimator being asymptotically Gaussian or the nuisance being estimated at the parametric rate.

In summary, Perturbed DML transforms the weakness of nuisance estimation (bias) into a strength by using perturbation to explore the space of possible estimates, filtering out implausible ones, and aggregating the rest to guarantee valid inference in difficult, high-dimensional settings.