Technical Summary: Upper-Expectile Multi-Step Q-Learning for Off-Policy Reinforcement Learning

Problem Statement

Off-policy value-based reinforcement learning (RL) aims to learn improved policies from logged experience collected by prior policies or human operators. While multi-step returns (n-step TD) accelerate reward propagation by shortening the effective bootstrap chain, they introduce a significant challenge in off-policy settings: pessimistic bias.

In standard n-step TD, the target for a state-action pair $(s_t, a_t)$ incorporates the rewards of the subsequent logged actions $a_{t+1:t+n-1}$, which are generated by a behavior policy $\pi_\beta$ rather than the target policy being learned. This couples the evaluation of the initial decision to the suboptimal continuations that follow it. On heterogeneous offline data, this induces a systematic pessimistic bias that grows with the horizon $n$, potentially degrading performance at long horizons.

Existing remedies face limitations:

• Importance Sampling (IS) variants (e.g., Retrace, V-trace): Require action likelihoods, which are often unavailable or expensive to compute for expressive generative policies (e.g., diffusion or flow-matching models).
• Long-Horizon Q-learning (LQL): Uses trajectory-level optimality inequalities to constrain value estimates. While effective, it requires evaluating critics at multiple positions within a trajectory and constructing pairwise constraints, increasing computational overhead.

The authors ask: Can the rapid propagation of n-step backups be retained while reducing off-policy bias, using only the two critic evaluations performed by a standard n-step update?

Methodology: Expectile n-step Q-learning (ENQ)

The paper proposes Expectile n-step Q-learning (ENQ), a modification of standard n-step TD that replaces the symmetric squared temporal-difference (TD) loss with an asymmetric expectile loss.

Core Mechanism

Instead of minimizing the squared error of the TD residual, ENQ minimizes the asymmetric expectile loss $\ell_\tau$ on the action-value error:
$$\ell_\tau(u) \triangleq |\tau - \mathbb{I}(u < 0)| u^2$$
where $\tau \in (0, 1)$ is the expectile level.

• Target Construction: For a segment of length $n$, the target $Y_{t,n}$ is constructed as the sum of the partial return $G_{t:t+n}$ and the bootstrapped value at $s_{t+n}$:
  $$Y_{t,n} = G_{t:t+n} + \gamma^n \bar{Q}^\rho_\theta(s_{t+n}, a^*(s_{t+n}))$$
  where $\bar{Q}^\rho_\theta$ is a conservative ensemble average of target critics.
• Loss Function: The critic $Q_\theta$ is trained to fit the $\tau$-expectile of the distribution of these n-step targets conditioned on $(s_t, a_t)$.
  $$L_{ENQ}(\theta) = \mathbb{E}_D \left[ v_t \cdot \frac{1}{K} \sum_{i=1}^K \ell_\tau(\delta_i) \right]$$
  where $\delta_i = \text{sg}[Y_{t,n}] - Q_{\theta,i}(s_t, a_t)$.

Key Design Choices

1. Asymmetric Weighting: For $\tau > 0.5$, positive residuals (underestimation) receive weight $\tau$, while negative residuals (overestimation) receive weight $1-\tau$. This forces the critic to track the upper tail of the logged n-step target distribution, effectively ignoring poor continuations caused by suboptimal behavior actions while retaining the multi-step reward signal.
2. Minimal Overhead: Unlike LQL, ENQ evaluates the critic only at the initial pair $(s_t, a_t)$ and the final bootstrap state $s_{t+n}$. It requires no intermediate evaluations, no action likelihoods, and no trajectory-level pairwise penalties.
3. Hyperparameters: The method introduces only one method-specific hyperparameter, the expectile level $\tau$, in addition to the standard backup horizon $n$.

Theoretical Contributions

The paper provides a rigorous theoretical analysis of the ENQ operator $T^n_\tau$:

1. Contraction Property: The ENQ operator is proven to be a $\gamma^n$-contraction for any $\tau \in (0, 1]$, ensuring the existence of a unique bounded fixed point $Q^n_\tau$ reachable by iteration.
2. Deterministic Dynamics Analysis:
   • Bias Decomposition: Under deterministic dynamics, the pessimism of standard n-step TD (at $\tau=0.5$) is exactly decomposed into a sum of discounted suboptimality gaps of the intervening logged actions.
   • Optimality at $\tau=1$: At $\tau=1$, the operator bias vanishes on covered in-support pairs (where the behavior policy places positive probability on optimal continuations). The fixed point $Q^n_1$ satisfies the lower-bound inequality used by LQL at separations $n, 2n, \dots$ by construction.
   • Monotonicity: The fixed point $Q^n_\tau$ is non-decreasing with respect to $\tau$ and is bounded above by the optimal value function $Q^*$.
3. Stochastic Dynamics Analysis:
   • The operator bias admits two-sided bounds with horizon-independent noise constants. The bias is bounded below by the behavioral drift (pessimism) and above by a term dependent on the variance of the returns and the expectile level $\tau$.
   • A "safe" threshold $\tau_{safe}$ is derived to ensure the operator bias remains non-positive (pessimistic) under stochastic dynamics.

Experimental Results

The authors evaluated ENQ across 27 task instances from OGBench and RoboMimic, covering manipulation and navigation domains.

Setup

• Baselines: Compared against 1-step TD, standard n-step TD, and Long-Horizon Q-learning (LQL).
• Configuration: Used a fixed $\tau = 0.8$ and $n=4$ across all tasks without task-specific tuning.
• Policy Extraction: Evaluated with Flow Q-Learning (FQL) and action-chunked variants.

Key Findings

1. Performance: ENQ achieved the higher aggregate mean success rate in 4 out of 5 comparisons against LQL. Notable gains were observed on complex domains like cube-triple, humanoid-md, and antmaze-giant.
2. Throughput: In a controlled profiling study with matched horizons and batch sizes, ENQ demonstrated 1.27x to 1.77x higher training throughput than LQL. This is attributed to ENQ's requirement for fewer critic evaluations per update (only start and end states vs. multiple points in a trajectory).
3. Ensemble Scaling: In a scaling experiment increasing the critic ensemble size from $K=2$ to $K=10$, ENQ benefited more significantly than LQL, reaching a 78.0% success rate compared to LQL's 59.1% on cube-triple.
4. Ablations:
   • Expectile Level: Upper expectiles ($\tau > 0.5$) generally outperformed the symmetric baseline ($\tau=0.5$). The optimal $\tau$ tended to increase with the horizon $n$.
   • Placement: Directly applying the expectile loss to the action-value error (ENQ) substantially outperformed an n-step IQL-style baseline that routed asymmetry through an auxiliary state-value function.
   • Robustness: Under high stochastic noise ($\sigma=0.25$), ENQ maintained superior performance (59.0% success) compared to LQL (5.4%) and standard n-step TD (48.4%).

Significance and Claims

The paper claims that ENQ offers a simple, flexible, and computationally efficient alternative to existing off-policy multi-step methods. Its significance lies in:

• Reducing Pessimism without Likelihoods: It mitigates the bias induced by suboptimal logged actions without requiring action likelihoods or importance weights, making it compatible with modern generative policy classes.
• Computational Efficiency: By avoiding trajectory-level constraints and intermediate critic evaluations, it achieves higher training throughput than constraint-based methods like LQL.
• Theoretical Grounding: It provides a clear theoretical link between the expectile level $\tau$ and the reduction of pessimistic bias, showing that the $\tau=1$ fixed point satisfies optimality inequalities similar to those enforced by LQL, but through a simpler operator.

The authors conclude that ENQ is a competitive, drop-in replacement for symmetric n-step TD that effectively balances reward propagation speed with bias reduction in off-policy reinforcement learning.