Technical Summary: Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Belief (PhyB)

1. Problem Statement

Offline Reinforcement Learning (RL) seeks to optimize policies using pre-collected datasets without further environmental interaction. A primary bottleneck in this paradigm is managing epistemic uncertainty, which arises from two sources:

1. Sample-level uncertainty: Caused by partial coverage of the dataset.
2. Model-level uncertainty: Arising from the ambiguity in identifying unique transition dynamics from finite data.

While Bayesian RL offers a unified framework to quantify these uncertainties by treating dynamics models as random variables, practical policy optimization remains computationally challenging. Existing approaches face a trade-off:

• Search-based techniques (e.g., Mixed-Integer Linear Programming, Monte Carlo Tree Search) are theoretically sound but fail to scale to high-dimensional tasks.
• Restrictive assumptions (e.g., fixed posterior distributions, robust MDPs with Dirac likelihoods) ensure tractability but sacrifice the adaptability and inherent performance of Bayesian RL, often leading to excessive conservatism (over-pessimism).

The core question addressed is: Can we develop a computationally tractable offline policy optimization algorithm under Bayesian beliefs without relying on restrictive assumptions on the posterior distribution?

2. Methodology: Posterior Hybrid Bayesian Belief (PhyB)

The authors propose PhyB, a framework that reformulates the expectation over model posteriors as a convex combination over a subset of dynamics models, avoiding restrictive posterior assumptions.

2.1. Belief Formulation

Instead of treating the dynamics model as a point estimate or maintaining a complex continuous posterior, PhyB operates on a finite model ensemble $\mathcal{T} = \{\tau_0, \dots, \tau_{N-1}\}$.

• Pessimistic Subset: A subset $\tilde{\mathcal{T}} \subseteq \mathcal{T}$ is formed by selecting the $k$ models corresponding to the lowest $Q$-value predictions (bottom-$k$).
• Convex Combination: Rather than selecting a single worst-case model (which leads to Dirac-like pessimism) or averaging all models (which dilutes pessimism), PhyB assigns weights $\alpha_i$ to models in $\tilde{\mathcal{T}}$.
• Entropy-Regularized Weighting: The weights are determined by solving an entropy-regularized optimization problem:
  $$\min_{\alpha} \sum_{i=0}^{k-1} \alpha_i q_{\tau_i}(s, a) + \lambda \alpha_i \log \alpha_i$$
  The optimal solution follows a Boltzmann distribution: $\alpha_i \propto \exp(-q_{\tau_i}(s, a)/\lambda)$.
• Hybrid Posterior: This weighting scheme constructs a "Posterior Hybrid Bayesian Belief" $\tilde{P}(\tau)$, which is a mixture of unimodal distributions centered around specific quantiles of the model ensemble. This allows for continuous, data-driven adaptation of the pessimism level.

2.2. Regularized Policy Optimization

To optimize the objective function defined by this hybrid belief, the authors address two challenges: the non-Euclidean geometry of the multimodal distribution space and the computational cost of solving the Bellman equation.

• Bregman-Regularized Subproblems: The optimization is decomposed into a sequence of subproblems regularized by Bregman divergence ($D_\psi$).
• Iterative Algorithm: An iterative policy improvement algorithm is proposed where the policy at step $i+1$ is updated based on the reference policy from step $i$:
  $$\pi_{i+1} = (\nabla \psi)^{-1} \left( \frac{1}{\beta} (Q - z) + \nabla \psi(\pi_i) \right)$$
  This formulation mirrors natural gradient methods and ensures monotonic improvement of the objective until convergence.

2.3. Theoretical Guarantees

• Bounded Discrepancy: The objective discrepancy induced by the convex combination approximation is proven to be bounded.
• Pessimism: The method induces a lower bound on the true expected return. The gap between the estimated and true return vanishes as model fidelity increases and the pessimistic subset size decreases.
• Monotonicity: The policy performance is proven to be monotonically non-decreasing with respect to the subset size $k$ and monotonically non-increasing with respect to the ensemble size $N$.

3. Key Contributions

1. PhyB Formulation: A novel belief formulation that treats transition dynamics as random variables without restrictive posterior assumptions, reformulating expectations as convex combinations over a pessimistic subset.
2. Iterative Regularized Optimization: An algorithm that decomposes the optimization task into Bregman-regularized subproblems, providing metric-agnostic guarantees for monotonic improvement.
3. Theoretical Analysis: Proofs establishing that the approximation error is bounded, the induced pessimism is controllable, and the policy optimization converges monotonically.
4. Empirical Performance: State-of-the-art results on standard benchmarks (D4RL) and stochastic liquidation tasks.

4. Experimental Results

The authors evaluated PhyB on the D4RL benchmark (18 datasets across HalfCheetah, Hopper, and Walker2d with varying data qualities) and a stochastic offline optimal liquidation task.

• Performance: PhyB achieved superior performance on 11 out of 18 D4RL datasets and remained competitive on the others. It notably outperformed strong baselines including model-free methods (CQL, IQL, TD3+BC) and model-based methods (MOReL, RAMBO, PMDB, ADM).
• Stochastic Tasks: In the highly stochastic liquidation task, where many baselines failed to learn effective policies, PhyB demonstrated robust and superior performance.
• Ablation Studies:
  • Convex Combination: Removing the convex combination (selecting a single model) caused significant performance degradation, validating the necessity of the hybrid belief.
  • Hyperparameters: The method showed robustness to hyperparameter variations ($\omega$, $\lambda$). A trade-off was observed where lower-quality datasets benefited from higher pessimism (smaller $\lambda$), while high-quality datasets required less conservatism.
• Theoretical Alignment: Empirical curves confirmed that PhyB's estimated returns consistently lower-bounded the true returns (validating the pessimism theorem) and exhibited near-monotonic improvement during training.

5. Significance and Claims

The paper claims that PhyB successfully bridges the gap between the theoretical appeal of Bayesian RL and the computational requirements of offline policy optimization.

• Unified Uncertainty Quantification: By treating dynamics as random variables and using a hybrid posterior, PhyB unifies sample-level and model-level uncertainty quantification without resorting to oversimplified assumptions like Dirac likelihoods.
• Scalability: Unlike prior Bayesian methods that rely on intractable solvers, PhyB offers a computationally efficient approach that scales to high-dimensional continuous control tasks.
• Adaptability: The method avoids the "excessive conservatism" of robust MDPs by allowing the pessimism level to adapt continuously based on the data and model disagreement, rather than relying on fixed worst-case assumptions.
• Robustness: The approach provides metric-agnostic guarantees for monotonic improvement, ensuring stable learning even in the absence of online environment interaction.

The authors conclude that PhyB represents a significant step forward in making Bayesian RL practical for offline settings, offering a robust framework that balances risk aversion with performance optimization.