Adaptive Off-Policy Inference for M-Estimators Under Model Misspecification

Technical Summary: Adaptive Off-Policy Inference for M-Estimators Under Model Misspecification

1. The Problem: Inference in Adaptive, Misspecified Settings

In modern data science, data is often collected via adaptive algorithms (e.g., multi-armed bandits, reinforcement learning). Unlike classical i.i.d. settings, adaptive data collection creates sequential dependencies: the choice of treatment at time $t$ depends on the history of previous rewards and actions. This leads to two major statistical hurdles:

• Unstable Variances: Common bandit algorithms (like UCB or Thompson Sampling) often result in non-convergent action probabilities, causing the variance of standard estimators to fluctuate wildly and preventing asymptotic normality.
• Model Misspecification: Most existing methods for adaptive inference assume the "working model" (the mathematical structure used to describe rewards) is correct. In practice, if the model is misspecified, the parameter being estimated may shift depending on the data-collection policy, making the target parameter ill-defined or biased.

The authors identify a gap: existing literature either assumes a correctly specified model or assumes that the adaptivity eventually "vanishes" (the policy converges). They address the much harder case where the model is wrong and the policy remains unstable/non-convergent.

2. Methodology: MAIPWM and Variance Stabilization

To solve this, the authors propose a framework for off-policy inference. Instead of trying to estimate the parameter of the adaptive policy, they target a parameter $\theta^*$ defined relative to a fixed, stationary evaluation policy $\pi_e$.

Key Methodological Components:

• Target Parameter ($\theta^*$): They define the target as the expected maximizer of a function $m_\theta$ under the evaluation policy $\pi_e$. This ensures the target remains stable even if the actual data-collection policy $\pi_t$ is erratic.
• MAIPWM-Estimator: The Misspecified Augmented Inverse Propensity Weighted M-estimator combines two ideas:
  1. Inverse Propensity Weighting (IPW): Weights observations by $\frac{\pi_e(a|x)}{\pi_t(a|x, H_{t-1})}$ to correct for the bias introduced by the adaptive policy.
  2. Augmentation (Doubly Robust style): Uses a predictive model $f_t$ (trained on history) to "augment" the score function, which improves statistical power by reducing variance.
• Variance Stabilization via Time-Varying Matrices: To achieve a valid Central Limit Theorem (CLT), the authors introduce a two-step procedure. They construct a sequence of matrices $\hat{V}_t$ that normalize the score function at each time step. Because $\hat{V}_t$ is unknown, they propose two practical ways to estimate it:
  • External Data: Using an independent dataset with the same feature distribution.
  • Sequential Sample Splitting: Splitting the adaptive stream into two parallel histories to ensure the variance estimate is independent of the parameter estimate.

3. Key Contributions

• Theoretical Breakthrough: The paper provides a Central Limit Theorem (Theorem 1) that holds under non-vanishing adaptivity and model misspecification. This allows for the construction of valid confidence intervals in settings where previous methods fail.
• Robustness to Policy Instability: Unlike prior work (e.g., Guo & Xu, 2025) that requires policies to converge, this method works even when treatment policies are unstable (e.g., when there is no unique optimal arm).
• Practical Algorithmic Framework: They provide a complete end-to-end algorithm (Algorithm 1) that integrates machine learning (Random Forests) to estimate conditional means and variances for Generalized Linear Models (GLMs).

4. Empirical Results

The authors tested their method using semi-synthetic data derived from the Osteoarthritis Initiative (OAI). They compared their method against Naive MLE, standard IPW, and SQ-IPW across various bandit strategies (Epsilon-Greedy, UCB, Thompson Sampling).

• Coverage Accuracy: In scenarios with model misspecification and heteroskedasticity, existing methods (IPW, SQ-IPW) failed to maintain the nominal coverage (e.g., a 90% confidence interval actually covered much less than 90% of the time). The proposed MAIPWM methods maintained correct coverage.
• Efficiency vs. Robustness: They demonstrated that while Sample Splitting provides valid coverage, it results in wider confidence intervals (a "price" for robustness). However, their "Reuse" approach (using the same data for both estimation and variance) performed nearly as well as the theoretically safer external data method.

5. Significance

This paper is significant because it bridges the gap between causal inference (which often assumes fixed interventions) and reinforcement learning (where interventions are adaptive). It provides a mathematically rigorous way for practitioners to perform "honest" statistical inference—meaning they can say how certain they are about a parameter—even when their underlying models are imperfect and their data collection is driven by complex, evolving algorithms. This has direct implications for clinical trials, personalized medicine, and automated recommendation systems.