Nonparametric Estimation of Optimal Stochastic Just-In-Time Adaptive Interventions for Distal Outcomes

Technical Summary: Nonparametric Estimation of Optimal Stochastic Just-In-Time Adaptive Interventions for Distal Outcomes

Problem Statement
Mobile and wearable technologies facilitate Just-In-Time Adaptive Interventions (JITAIs), which adapt treatment delivery to an individual's real-time state. While Micro-Randomized Trials (MRTs) generate the intensive longitudinal data necessary to construct these interventions, existing methods face significant challenges when targeting distal outcomes (e.g., end-of-study outcomes) rather than proximal, immediate responses.

Standard longitudinal causal inference methods, such as Marginal Structural Models (MSMs), often struggle in settings with a large number of decision points ($T$). Specifically:

1. Instability: Inverse probability weighting (IPW) estimators involve products of weights across $T$ time points, leading to numerical instability, high variance, and substantial finite-sample bias when $T$ is large.
2. Irregularity: Doubly robust estimators require estimating a large collection of augmentation terms (outcome regressions indexed by time and policy), which becomes impractical and can lead to irregular estimators with slow convergence rates.
3. Limitations of Reinforcement Learning (RL): While RL is designed for sequential decision-making, it typically optimizes cumulative or discounted proximal rewards. Selecting an appropriate discount rate for distal outcomes is often empirically unsupported, and RL methods may lack interpretability.
4. Positivity Issues: Deterministic or near-deterministic policies often suffer from finite-sample positivity violations in high-dimensional decision spaces, where specific treatment paths are poorly supported in the observed data.

Methodology
The authors propose a unified framework for estimating and optimizing stochastic JITAIs for distal outcomes using a nonparametrically efficient estimator. The core components are:

1. Stochastic Policy Class: The authors focus on a parametric class of stochastic policies $\{q_\theta\}$, where treatment probabilities are functions of the current state $S_t$ (modeled via logistic regression with basis functions). Stochastic policies are preferred over deterministic ones to maintain flexibility, incorporate unmodeled constraints, and mitigate positivity issues.

2. $\alpha$-Tilting Device: To stabilize estimation in high-$T$ settings, the authors introduce a data-adaptive tilting procedure. This interpolates between the target policy $q_\theta$ and the observed treatment mechanism $\pi_{t,0}$ at each decision point $t$:
   $$q_{\theta, \alpha, t} = (t/T)^\alpha q_\theta + \{1 - (t/T)^\alpha\}\pi_{t,0}$$
   For $\alpha > 0$, early decision weights are shrunk toward 1, reducing the volatility of the product of inverse probability weights. The parameter $\alpha$ is selected via a data-adaptive rule that minimizes the estimated Mean Squared Error (MSE) of the value function, balancing bias (drift from the scientific target) and variance (stabilization).

3. Undersmoothed Inverse Probability Weighting (UIPW) Estimator:

   • Instead of estimating the complex, policy-indexed outcome regression functions required for efficient estimation, the authors utilize the Highly Adaptive Lasso (HAL).
   • They employ undersmoothing (selecting a penalty parameter $\lambda$ larger than that chosen by cross-validation) to ensure the fitted propensity score models are rich enough to approximately solve the empirical score equations associated with the efficient influence function.
   • This approach achieves nonparametric efficiency ($\sqrt{n}$-consistency) without explicitly estimating the nuisance outcome regressions, thereby avoiding the "curse of dimensionality" associated with the augmentation terms.

4. Penalized Optimization: To avoid boundary solutions (which can be numerically unstable), the optimal policy parameter $\theta$ is found by maximizing a penalized value function $V_n(\theta) = \psi_n(\theta) - P(\theta)$, where $P(\theta)$ is an inverse-distance barrier-weighted ridge penalty that pushes the optimizer away from the boundary of the parameter space $\Theta$.

Key Contributions

1. Nonparametrically Efficient Estimator: Development of a UIPW estimator for the regimen-response curve (expected distal outcome under a stochastic policy) that attains the nonparametric efficiency bound without estimating the full set of time- and policy-indexed outcome regressions.
2. Data-Adaptive Stabilization: Introduction of a tilting procedure that stabilizes estimation in high-$T$ settings by shrinking early weights, with theoretical guarantees that the bias introduced by tilting is asymptotically negligible if $\alpha_n \to 0$ sufficiently fast.
3. Asymptotic Theory for Inference:
   • Proof that the estimated regimen-response curve converges weakly to a Gaussian process, enabling the construction of simultaneous confidence bands over the policy parameter space.
   • Derivation of the asymptotic distribution of the optimizer $\theta^*_n$, enabling valid inference for the learned optimal stochastic policy.
4. Practical Implementation: Proposals for data-adaptive selectors for the undersmoothing level (based on DCAR norms or score residuals) and the tilting parameter $\alpha$.

Results
The authors evaluate their method through numerical simulations mimicking mobile health trials (e.g., smoking cessation) with varying sample sizes ($n$), decision points ($T$), and treatment assignment mechanisms (MRT vs. observational).

• Bias Control: In observational settings where the treatment mechanism must be learned, standard cross-validated HAL estimators exhibited significant bias as $T$ increased. The UIPW estimators with undersmoothing successfully controlled this bias.
• Stability and MSE: Adaptive tilting reduced the Mean Squared Error (MSE) for nearly all estimators, particularly in settings with large $T$ where weight accumulation causes instability.
• Policy Optimization: The adaptive tilting approach generally improved the Root Mean Squared Error (RMSE) of the estimated optimal policy compared to untilted estimators.
• Coverage: While standard estimators showed undercoverage in regions where the target policy differed significantly from the observational policy (due to numerical positivity issues), the proposed methods maintained coverage rates closer to nominal levels, though some overcoverage was observed in specific settings.
• Selector Performance: Among the undersmoothing selectors, the $L_2$ DCAR-based selector generally provided more stable performance and lower regret than the score-based selector, which sometimes oversmoothed aggressively.

Significance and Claims
The paper claims to provide a unified framework for the estimation, inference, and optimization of stochastic JITAIs specifically for distal outcomes.

• Bridging the Gap: The work addresses the methodological gap between the structure of MRT data (many decision points) and the inferential targets (distal outcomes) that define long-term success in behavioral health.
• Feasibility: By avoiding the explicit estimation of high-dimensional outcome regressions, the method makes nonparametric efficient estimation feasible in intensive longitudinal settings where previous methods were computationally or statistically intractable.
• Robustness: The combination of stochastic policies, tilting, and undersmoothing offers a robust solution to the instability and positivity problems inherent in high-frequency decision-making data.
• Inference: The derivation of asymptotic properties for both the curve and the optimizer allows researchers to not only find an optimal policy but also quantify uncertainty through simultaneous confidence bands and policy-specific inference, a capability often lacking in current RL or policy-learning approaches.

The authors remain modest, noting that while undersmoothing is theoretically necessary for efficiency, its finite-sample performance depends heavily on the selection of tuning parameters. They also acknowledge that efficiency gains from nonparametric methods may be smaller in stochastic settings compared to deterministic ones due to the inherent stability of probabilistic weights.