Distributional Off-Policy Evaluation with Deep Quantile Process Regression

1. Problem Statement

The paper addresses the Off-Policy Evaluation (OPE) problem in Reinforcement Learning (RL). OPE aims to estimate the performance of a target policy ($\pi$) using data collected from a different behavior policy ($\pi_b$).

• Limitation of Existing Methods: Traditional OPE methods focus solely on estimating the expected value (mean) of the total return. However, in complex, stochastic environments (e.g., healthcare, robotics), the mean is often insufficient to capture risk, uncertainty, or the full complexity of decision-making.
• The Gap: While Distributional Reinforcement Learning (DRL) models the entire return distribution, existing DRL-based OPE methods often rely on discrete quantile approximations (e.g., QR-DQN). These methods suffer from:
  1. Representation Error: Discretizing the distribution into fixed quantile levels introduces approximation errors.
  2. Pseudo-Sample Issues: Generating target samples for the next iteration often requires "pseudo-samples" or imputation steps, which are computationally expensive and fail to perfectly reconstruct the target distribution.
  3. Theoretical Limitations: There is a lack of rigorous sample complexity analysis for distributional OPE using deep neural networks (DNNs) in non-tabular settings.

2. Methodology: DQPOPE

The authors propose Deep Quantile Process regression-based Off-Policy Evaluation (DQPOPE). This method shifts the paradigm from estimating discrete quantiles to estimating a continuous quantile function (a quantile process).

Core Concepts

• Quantile Process Regression: Instead of learning a set of discrete quantiles $\{f_{\tau_1}, \dots, f_{\tau_m}\}$, the model learns a single function $f(s, a, \tau)$ where $\tau \in (0, 1)$ is treated as a continuous input. This function maps a state-action pair and a quantile level to the corresponding return value.
• Distributional Bellman Update: The method leverages the one-to-one correspondence between a distribution and its quantile function. The distributional Bellman operator $T^\pi$ is reformulated as a regression problem on the quantile function.
• Exact Sampling: A key innovation is the ability to generate exact samples for the target variable $Y_t = R + \gamma Z_{t-1}(S', A')$ without pseudo-samples. By sampling a uniform random variable $U \sim \text{Unif}(0, 1)$ and plugging it into the learned quantile function $f_{t-1}(S', A', U)$, the model generates a sample with the exact same distribution as the future return.

Algorithm Overview

1. Initialization: Initialize a Deep Neural Network (DNN) $f_0$ representing the quantile function.
2. Iterative Update: For $t = 1$ to $T$:
   • Collect transitions $(s_i, a_i, r_i, s'_i)$ from the dataset.
   • Sample a quantile level $\tau_i \sim \text{Unif}(0, 1)$.
   • Sample an action $a'_i \sim \pi(\cdot|s'_i)$ and a uniform variable $u_i \sim \text{Unif}(0, 1)$.
   • Construct the target value: $y_i = r_i + \gamma f_{t-1}(s'_i, a'_i, u_i)$.
   • Update the network by minimizing the empirical Check Loss (quantile loss):
     $$\hat{L}_t(f) = \frac{1}{|D_t|} \sum_{i} \rho_{\tau_i}(y_i - f(s_i, a_i, \tau_i))$$
     where $\rho_\tau(u) = u(\tau - \mathbb{1}_{u \le 0})$.

3. Key Contributions

Theoretical Contributions

• Sample Complexity Analysis: The paper provides the first rigorous non-asymptotic statistical guarantees for distributional OPE using DNNs.
  • Slow Rate: Under standard assumptions, the excess risk converges at a rate of $N^{-\frac{\beta}{2\beta+d}}$ (where $\beta$ is smoothness, $d$ is dimension).
  • Fast Rate: By introducing a local strong convexity and local smoothness assumption (Assumption 4.10), the authors derive a fast rate of $N^{-\frac{\beta}{2\beta+d}}$.
• Equivalence of Efficiency: Crucially, the analysis demonstrates that estimating the entire return distribution via DQPOPE requires the same sample complexity as estimating only the mean value using standard value-based OPE. This proves that distributional methods do not incur a "sample penalty" compared to mean-based methods.
• Resolution of Pseudo-Sample Issue: Theoretically validates that quantile process regression eliminates the need for the "pseudo-sample" construction used in prior QDRL methods, thereby avoiding representation errors.

Practical Contributions

• Robustness to Heavy Tails: By using quantile loss (which is less sensitive to outliers than squared loss) and averaging over multiple quantiles to estimate the mean, DQPOPE provides more robust policy value estimates in environments with heavy-tailed reward distributions.
• Continuous Representation: The method captures the full continuous distribution, avoiding the discretization artifacts inherent in categorical or fixed-quantile approaches.

4. Experimental Results

The authors validated DQPOPE across three scenarios:

1. CartPole Simulation:

   • Demonstrated that estimation error decreases exponentially with sample size, aligning with theoretical predictions.
   • Showed that DQPOPE accurately recovers the ground-truth return distribution even under significant distributional shift (off-policy data).

2. Toy Example (Heavy-Tailed Rewards):

   • Compared DQPOPE against a standard value-based OPE (DOPE) using Deep ReLU networks.
   • Result: DQPOPE consistently achieved lower Mean Squared Error (MSE) in policy value estimation, especially as the reward distribution became more heavy-tailed (Student's t-distribution). Increasing the number of quantiles ($K$) used for averaging further improved accuracy.

3. Real-World Application (MIMIC-III Sepsis Dataset):

   • Applied to a clinical decision-making problem (treating sepsis) using 40,000+ patient records.
   • Result: DQPOPE outperformed both standard value-based OPE (DOPE) and discrete quantile-based OPE (DQOPE/QR-DQN).
   • Policy Quality: The policies estimated by DQPOPE were more aligned with the optimal target policy (DDQN) and produced clinically sensible treatment strategies (e.g., avoiding unnecessary high doses of vasopressors), whereas other methods produced suboptimal or erratic policies.

5. Significance and Impact

• Bridging Theory and Practice: The paper successfully bridges the gap between the theoretical advantages of distributional RL and practical implementation, providing a mathematically sound framework for distributional OPE.
• Efficiency: It challenges the intuition that modeling full distributions is more data-hungry than modeling means, proving they can be equally sample-efficient.
• Robustness: The method offers a superior solution for high-stakes, uncertain environments (like healthcare) where understanding the full risk profile (distribution) is as important as the expected outcome.
• Algorithmic Improvement: By resolving the "pseudo-sample" issue, DQPOPE offers a more stable and efficient training procedure for distributional RL algorithms compared to existing state-of-the-art methods.

In summary, DQPOPE represents a significant advancement in off-policy evaluation, offering a theoretically grounded, sample-efficient, and empirically robust method for estimating the full return distribution in complex, real-world reinforcement learning tasks.