🤖 machine learning

Bayesian policy gradient and actor-critic algorithms

This paper proposes a Bayesian framework for policy gradient and actor-critic algorithms that models gradients and action-value functions using Gaussian processes to reduce sample complexity, provide uncertainty estimates, and achieve closed-form posterior updates, thereby outperforming conventional Monte-Carlo methods in various reinforcement learning tasks.

Original authors: Mohammad Ghavamzadeh, Yaakov Engel, Michal Valko

Published 2026-05-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mohammad Ghavamzadeh, Yaakov Engel, Michal Valko

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to walk, or a video game character to navigate a maze. The robot doesn't know the rules of the world; it only knows what happens when it takes an action (like "step forward" or "turn left"). This is called Reinforcement Learning.

The goal is to find the best set of instructions (a "policy") that gets the robot to its goal as efficiently as possible. To do this, the robot needs to know which direction to tweak its instructions to get better. This direction is called the gradient.

The Old Way: Guessing in the Dark

Traditionally, robots figure out this direction using a method called Monte-Carlo. Imagine you are trying to find the best route through a foggy forest. The old way is to send out 1,000 explorers, have them all walk random paths, and then ask, "Who got the furthest?" You average their results to guess which way is "uphill."

The problem? It's incredibly noisy. One explorer might get lucky and find a shortcut, while another trips over a root. To get a reliable answer, you need thousands of explorers, which takes a long time and wastes a lot of energy (data).

The New Idea: The Bayesian "Smart Map"

This paper proposes a smarter way called Bayesian Policy Gradient. Instead of just guessing based on raw data, the robot builds a Smart Map (using something called a Gaussian Process) of how its instructions affect its success.

Think of it like this:

The Old Way: You ask 1,000 people for directions and take the average.
The New Way: You ask 10 people, but you also use your prior knowledge of the terrain (the map) to fill in the gaps. You know that if a path goes uphill for a bit, it likely continues uphill. You don't need 1,000 people to tell you that; 10 people plus your map is enough.

This "Smart Map" allows the robot to learn the correct direction with far fewer samples. It also tells the robot how confident it is in that direction (the uncertainty). If the map is blurry, the robot knows to be careful; if the map is clear, it can move fast.

Two Approaches to the Problem

The paper introduces two specific ways to build this Smart Map:

1. The "Whole Journey" Approach (Bayesian Policy Gradient)

Imagine you are a travel agent. In this approach, you look at the entire trip a traveler took from start to finish. You ask, "Did this whole journey work well?"

The Good News: This works even if the world is chaotic or if the traveler can't see everything (like driving in heavy fog). You don't need to know the exact rules of the road; you just look at the final result of the trip.
The Bad News: Because you are looking at the whole trip as one big block, you miss out on the little details that happen step-by-step. It's less efficient if the world does follow clear, predictable rules (like a standard video game level).

2. The "Step-by-Step" Approach (Bayesian Actor-Critic)

This is a more advanced method. Imagine you have a Coach (the Actor) and a Judge (the Critic).

The Coach decides what move to make.
The Judge watches every single step the Coach makes and gives immediate feedback: "That was a good step," or "That was a bad step."
The Judge uses a "Smart Map" to predict the value of every single move, not just the final outcome.

Because the Judge looks at every single step (state-action-reward), this method is much more efficient when the world follows predictable rules. It learns faster and with less data than the "Whole Journey" approach.

What Did They Prove?

The authors ran experiments to see if their "Smart Map" methods actually worked better than the old "Guessing in the Dark" methods. They tested them on:

Simple games: Like a slot machine (Bandit problem).
Control tasks: Like balancing a pole or steering a ship.

The Results:

The new methods learned much faster and with less data than the old methods.
The "Step-by-Step" (Actor-Critic) method was the most efficient, especially in predictable environments.
The methods were also able to handle situations where the robot couldn't see the whole picture (Partially Observable problems), which is a common real-world issue.

In Summary

This paper is about teaching robots to learn more efficiently. Instead of blindly trying thousands of random actions to figure out what works, the authors gave the robots a "Smart Map" (Bayesian inference) that helps them understand the world with fewer tries. They showed that by combining this map with a "Coach and Judge" system, robots can learn complex tasks much faster and more reliably than before.

Technical Summary: Bayesian Policy Gradient and Actor-Critic Algorithms

1. Problem Statement

Policy Gradient (PG) methods are a cornerstone of Reinforcement Learning (RL), optimizing a parameterized policy by following an estimate of the performance gradient. However, conventional PG methods rely heavily on Monte-Carlo (MC) techniques to estimate these gradients.

High Variance: MC estimators are unbiased but suffer from high variance, requiring a large number of samples to achieve accurate estimates, which leads to slow convergence.
Sample Inefficiency: The reliance on frequentist point estimates ignores the uncertainty inherent in the data, failing to leverage prior knowledge or the structure of the data distribution effectively.
Limitations of Existing Solutions: While baselines and natural gradients (using the Fisher Information Matrix) help, they do not fundamentally address the variance issue or the lack of uncertainty quantification in the gradient estimate itself.

The paper addresses the need for a more sample-efficient, robust, and uncertainty-aware framework for policy gradient estimation and actor-critic learning.

2. Methodology

The authors propose a Bayesian framework that replaces the frequentist MC estimation with Bayesian Quadrature (BQ). The core idea is to model the gradient of the expected return as a Gaussian Process (GP).

A. Bayesian Policy Gradient (BPG)

The BPG approach treats the gradient integral $\nabla \eta(\theta) = \int \bar{R}(\xi) \nabla \log P(\xi; \theta) d\xi$ as an integral of a random function.

Modeling: The integrand is split into a deterministic part (related to the policy probability) and a stochastic part (the return). The stochastic part is modeled as a GP.
Two Models:
1. Model 1 (Vector-Valued GP): Models the gradient components directly as a vector-valued GP. It uses a quadratic Fisher kernel.
2. Model 2 (Scalar-Valued GP): Models the return as a scalar GP and integrates it against the gradient of the policy probability. It uses a standard Fisher kernel.
Key Mechanism: By observing trajectories, the algorithm computes the posterior distribution of the gradient. This yields not just a point estimate (the mean) but also a covariance matrix representing the uncertainty of the gradient estimate.
Kernel Choice: The use of Fisher Kernels (based on the Fisher Information Matrix) is crucial. It allows for the derivation of closed-form expressions for the posterior moments of the gradient, making the computation tractable.
Applicability: Since BPG treats entire system trajectories as the basic unit, it does not require the Markov property, making it naturally extendable to Partially Observable MDPs (POMDPs).

B. Bayesian Actor-Critic (BAC)

To address the limitation that BPG cannot exploit the Markov property (leading to higher variance in Markovian systems), the authors introduce a Bayesian Actor-Critic algorithm.

Critic: Uses Gaussian Process Temporal Difference (GPTD) learning. The action-value function $Q(z)$ is modeled as a GP.
Actor: Updates policy parameters using the posterior moments of the gradient derived from the critic's GP.
Integration: By choosing a specific prior covariance kernel for the GP (sum of a state kernel and the Fisher kernel), the authors derive closed-form expressions for the posterior mean and covariance of the policy gradient.
Advantage: BAC uses individual state-action-reward transitions as the observable unit. This allows it to exploit the Markov property, significantly reducing variance compared to trajectory-based BPG, especially in long-horizon problems.

C. Online Sparsification

Both BPG and BAC involve kernel matrices that grow with the number of samples ( $O(M^3)$ complexity). The paper incorporates online sparsification (based on Engel et al., 2002) to maintain a "dictionary" of representative samples. This ensures the algorithms remain computationally efficient in time and memory while controlling approximation error via a threshold parameter.

3. Key Contributions

Bayesian Policy Gradient Framework: A novel formulation of PG methods using Bayesian Quadrature and Gaussian Processes, providing a full posterior distribution over the gradient rather than a single point estimate.
Closed-Form Solutions: The derivation of analytical expressions for the posterior mean and covariance of the gradient using Fisher kernels, avoiding the need for numerical integration or sampling for the gradient estimation itself.
Uncertainty Quantification: The ability to compute the gradient covariance at little extra cost. This allows for risk-aware learning (e.g., using the covariance to adapt step sizes).
Bayesian Actor-Critic (BAC): A new AC algorithm that combines GPTD critics with Bayesian policy updates, leveraging the Markov property for lower variance than BPG.
Extension to POMDPs: Demonstrating that the trajectory-based BPG approach naturally handles non-Markovian systems without modification.
Comprehensive Evaluation: Extensive experimental comparisons against classic MC-based PG and Actor-Critic methods across diverse domains.

4. Experimental Results

The authors evaluated their methods on several benchmarks:

Bandit Problem & Linear Quadratic Regulator (LQR):
- BQ (Bayesian Quadrature) estimates of the gradient showed orders of magnitude lower variance and higher accuracy compared to MC estimates, particularly with small sample sizes.
- BPG (using natural gradients) outperformed MCPG (Monte-Carlo PG) in LQR, converging faster and achieving better final returns, especially with moderate sample sizes ( $M=10, 20, 40$ ).
- A variant using gradient covariance to adjust step sizes (BPG-var) showed the best performance.
Random Walk (10-state):
- BAC significantly outperformed both BPG and MCPG. BAC learned faster and was more robust to learning rate changes.
- BPG struggled when trajectory lengths varied, whereas BAC (using state-action transitions) remained stable.
Mountain Car (Continuous State/Discrete Action):
- BAC consistently outperformed MCPG, finding good policies with very few samples ( $M=5$ ) and maintaining low variance. MCPG required larger sample sizes to match BAC's performance.
Ship Steering (Continuous State/Continuous Action):
- In this complex, high-dimensional domain, BAC again demonstrated superior convergence speed and success rates compared to MCPG, with lower performance variance.

5. Significance and Impact

Sample Efficiency: The primary significance is the drastic reduction in the number of samples required to learn an effective policy. By modeling the gradient as a GP, the algorithm "learns" the shape of the gradient function, allowing it to generalize from fewer data points than MC methods.
Uncertainty-Aware Learning: The provision of gradient covariance enables risk-sensitive policy updates. The algorithm can take larger steps when uncertainty is low and smaller, safer steps when uncertainty is high, a feature absent in standard PG methods.
Theoretical Rigor: The paper provides rigorous proofs for the closed-form solutions of the posterior moments, bridging the gap between Bayesian non-parametric methods and RL.
Versatility: The framework unifies the handling of both Markovian (via BAC) and non-Markovian/POMDP (via BPG) problems under a single Bayesian umbrella.

In conclusion, this work establishes a strong theoretical and empirical foundation for Bayesian Reinforcement Learning, demonstrating that treating policy gradients as random variables modeled by Gaussian Processes leads to more efficient, stable, and robust learning algorithms compared to traditional frequentist approaches.