Inference for relative sparsity

1. Problem Statement

The paper addresses the challenge of developing statistical inference (e.g., confidence intervals, hypothesis testing) for relative sparsity in the context of estimating optimal treatment policies from observational data.

• Context: In healthcare, there is a need to derive treatment policies (mappings from patient covariates to actions) that improve upon the "standard of care" (the behavioral policy that generated the data). However, purely maximizing reward often leads to policies that are too complex or diverge too far from established guidelines, hindering clinical adoption.
• The Relative Sparsity Approach: Previous work (Weisenthal et al., 2023) introduced a relative sparsity penalty to encourage policies that differ from the standard of care only on a sparse set of covariates, thereby enhancing explainability.
• The Gap: The original relative sparsity framework focused solely on estimation, not inference. Furthermore, it was limited to single-stage decisions.
• Key Challenges for Inference:
  1. Infinite Estimands: In binary action settings with parameterized policies (e.g., logistic regression), the optimal policy is deterministic. To approximate a deterministic step function using a smooth link function (like expit), the coefficients must diverge to infinity, making standard inference impossible.
  2. Instability: The raw value function (expected reward) in multi-stage settings is highly unstable due to the inverse probability weighting (IPW) ratio, which can have infinite variance.
  3. Non-differentiability: The Lasso-type penalty used for sparsity is non-differentiable, complicating asymptotic theory.
  4. Post-Selection Bias: Selecting variables (covariates) based on the data and then performing inference on the same data leads to biased confidence intervals.

2. Methodology

The authors propose a novel framework that integrates Trust Region Policy Optimization (TRPO), Adaptive Lasso, and Sample Splitting to overcome these challenges.

A. Double Behavior Constraint (Solving Infinite Estimands)

To prevent coefficients from diverging to infinity, the authors replace the raw value objective with a weighted TRPO objective as the "base" for the relative sparsity penalty.

• Base Objective ($M_0$): They maximize a value function penalized by a Kullback-Leibler (KL) divergence from the behavioral policy:
  $$M_0(\beta, b, \gamma) = V_0(\beta) - \gamma KL_0(\beta, b)$$
  where $\gamma$ controls the strength of the behavior constraint. This ensures the resulting estimator $\beta_{0,\gamma}$ is finite and amenable to inference.
• Relative Sparsity Objective ($W_0$): They add an adaptive Lasso penalty to the TRPO objective to enforce sparsity relative to the behavioral policy:
  $$W_0(\beta) = M_0(\beta) - \lambda \sum_{k=1}^K w_{0,k} |\beta_k - b_{0,k}|$$
  where $w_{0,k} = 1/|\beta_{0,\gamma,k} - b_{0,k}|^\delta$ (Adaptive weights from Zou, 2006).

B. Sample Splitting Framework

To address post-selection inference issues, the authors employ a sample-splitting strategy:

1. Split 1 (Selection): Used to estimate the behavioral policy ($b_n$), the base TRPO policy ($\beta_{n,\gamma}$), and to select the active set of covariates ($A$) using the adaptive penalty.
2. Split 2 (Inference): Used to perform inference only on the non-behavioral coefficients in the selected set $A$.
   • Novelty: Unlike standard sample splitting where non-selected variables are dropped, this framework fixes non-selected coefficients to their behavioral counterparts ($b_n$) within the suggested policy. This creates a complex dependency where the nuisance parameter $b$ appears in both the behavioral and suggested policies, requiring new derivations for gradients and Hessians.

C. Estimation and Tuning

• Value Estimation: Uses a self-normalized (weighted) IPW estimator to handle multi-stage instability.
• Tuning Parameters:
  • $\gamma$: Controls closeness to the standard of care (stability).
  • $\lambda$: Controls the degree of sparsity. Selected such that the policy achieves a value improvement of at least one standard error over the behavioral policy while maximizing sparsity.
  • $\delta$: Controls the adaptivity of the Lasso weights.

3. Key Contributions

1. Theoretical Foundation for TRPO Inference: The paper provides the first rigorous asymptotic theory for weighted TRPO estimators, proving consistency and asymptotic normality ($\sqrt{n}(\beta_{n,\gamma} - \beta_{0,\gamma}) \xrightarrow{d} N(0, \Sigma)$).
2. Solving the Infinite Estimand Problem: By nesting a KL-divergence constraint within the relative sparsity framework, they create finite, behavior-constrained estimands that allow for valid statistical inference in binary action settings.
3. Adaptive Relative Sparsity: They extend the adaptive Lasso to the relative sparsity setting, improving the ability to distinguish between coefficients that truly differ from the standard of care and those that do not.
4. Post-Selection Inference Framework: They develop a novel sample-splitting approach where non-selected parameters are fixed to behavioral estimates rather than zero, deriving the necessary asymptotic theory for this specific "nuisance-in-the-policy" scenario.
5. Multi-Stage Generalization: The methodology is extended from single-stage to general Multi-Stage Markov Decision Processes (MDPs), which is critical for dynamic treatment regimes in medicine.

4. Results

The authors validate their approach through extensive simulations and a real-world data analysis.

• Simulations:

  • Coverage: The proposed confidence intervals achieve nominal coverage (approx. 95%) for active parameters when tuning parameters ($\gamma$) are chosen to ensure objective stability.
  • Stability: The simulations demonstrate that small $\gamma$ leads to unstable estimates (wide confidence intervals), while larger $\gamma$ stabilizes inference at the cost of some value gain.
  • Selection: The method successfully identifies the correct sparse set of covariates while maintaining valid inference for the selected coefficients.

• Real Data Analysis (MIMIC-III):

  • Dataset: Applied to vasopressor administration for hypotension in ICU patients.
  • Outcome: The method derived a relatively sparse policy where only the Mean Arterial Pressure (MAP) covariate differed significantly from the standard of care.
  • Inference: Unlike previous estimation-only studies, the authors provided a 95% confidence interval for the MAP coefficient in the suggested policy ($0.132, [-0.136, 0.400]$), showing it was shifted toward a positive range compared to the behavioral policy, offering a statistically rigorous justification for the change in treatment strategy.

5. Significance

This work bridges a critical gap between reinforcement learning (RL) and causal inference in healthcare.

• Clinical Adoption: By providing confidence intervals and hypothesis tests, the method allows clinicians to assess the uncertainty of a new policy, which is a prerequisite for adopting data-driven strategies in high-stakes medical environments.
• Explainability: The relative sparsity constraint ensures that new policies are interpretable (differing from standard care on few variables), and the inference framework validates which differences are statistically significant.
• Methodological Rigor: It establishes a new standard for inference in policy search problems, particularly those involving non-differentiable penalties and behavior-constrained objectives, moving beyond "black box" prediction to rigorous decision-making.

In summary, the paper transforms relative sparsity from a purely optimization-based tool into a statistically rigorous framework capable of supporting evidence-based clinical decision-making.