⚡ electrical engineering

A Pontryagin Method of Model-based Reinforcement Learning via Hamiltonian Actor-Critic

This paper introduces Hamiltonian Actor-Critic (HAC), a model-based reinforcement learning method that leverages the Pontryagin Maximum Principle to directly optimize a Hamiltonian over learned dynamics, thereby eliminating explicit value function learning to achieve superior sample efficiency, robustness to model errors, and state-of-the-art performance in both online and offline continuous control tasks.

Original authors: Chengyang Gu, Yuxin Pan, Hui Xiong, Yize Chen

Published 2026-04-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chengyang Gu, Yuxin Pan, Hui Xiong, Yize Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Teaching a Robot to Walk Without a Map

Imagine you are trying to teach a robot to walk across a room. You have two main ways to do this:

Trial and Error (The Old Way): You let the robot walk, it falls down, you say "ouch," it tries again. It does this millions of times. This is slow and wastes a lot of energy.
Learning a Model (The New Way): You teach the robot how physics works first. "If I push my leg forward, I move forward." Then, you let the robot imagine walking in its head a thousand times before actually moving. This is much faster. This is called Model-Based Reinforcement Learning (MBRL).

The Problem:
The "imagination" part is tricky. If the robot's internal model of physics is slightly wrong (e.g., it thinks the floor is slippery when it's not), and it imagines walking for 100 steps, that small error gets magnified. By step 100, the robot thinks it's floating in space!

In standard AI methods, the robot also has a "Critic" (a judge) that tries to guess how good a move is. But if the robot's imagination is wrong, the Judge gets confused, gives bad scores, and the robot learns the wrong lessons. This is like a student trying to study for a math test using a textbook with typos; the more they study, the more confused they get.

The Solution: The "Hamiltonian" Compass

The authors of this paper propose a new method called Hamiltonian Actor-Critic (HAC).

Instead of using a "Judge" (Critic) to guess the score of a move, they use a concept from physics called the Hamiltonian.

The Analogy: The Hiker and the Compass

The Old Way (Critic): Imagine a hiker trying to find the top of a mountain. They ask a local guide (the Critic), "Is this path good?" The guide looks at a map (the model), but the map has some smudges. The guide guesses the height of the mountain based on the smudged map. If the map is wrong, the guide gives bad directions.
The New Way (HAC): Instead of asking a guide, the hiker uses a Compass (the Hamiltonian). The compass doesn't guess the height of the mountain; it simply points toward the steepest, most efficient path based on the laws of physics (gravity, friction, etc.).

The Hamiltonian is a mathematical formula that combines where you are, what you are doing, and where you want to go. It tells the robot exactly how to move to be most efficient right now, without needing to guess the total score of the whole journey.

How It Works (The Magic Trick)

No More Guessing: The robot doesn't need to train a separate "Judge" network to guess future rewards. It just calculates the "Compass reading" (the Hamiltonian) directly using its learned model of the world.
The "Backward" Look: To use this compass, the robot does something clever. It imagines a path forward, but then it calculates the "cost" of that path by working backward from the finish line to the start. This is like planning a road trip by starting at your destination and figuring out how to get there, rather than guessing what traffic will be like 5 hours from now.
The "Jacobian" Shortcut: Calculating this backward path usually requires heavy math that slows down computers. The authors invented a "shortcut" (Analytical Jacobian) that lets the computer do these calculations instantly, like using a pre-calculated formula instead of doing long division every time.

Why Is This Better?

The paper proves two main things:

It's More Accurate: Because the robot isn't relying on a "Judge" that might be biased by bad imagination, the errors don't pile up. It's like the difference between a GPS that recalculates your route every second based on real traffic (HAC) versus one that guesses traffic based on a 10-year-old map (Old Methods).
It's More Robust: If you put the robot in a slightly different room (a new environment), the old methods often panic and fail because their "Judge" has never seen this before. The HAC method, relying on the laws of physics, adapts much better. It's like a hiker who knows how to use a compass can find their way even if the trail markers are missing, whereas someone relying on a guide might get lost.

The Results

The researchers tested this on various tasks:

Simple tasks: Like balancing a pole (Pendulum).
Hard tasks: Like making a robot hop (Hopper) or swim (Swimmer).

The Verdict:

Faster Learning: The robot learned to do the tasks in fewer tries (better sample efficiency).
Better Performance: It got higher scores than the previous best methods.
Stability: Even when the data was limited or the starting position changed, it didn't crash as often as the other methods.

In a Nutshell

This paper introduces a smarter way for robots to learn. Instead of asking a fallible "Judge" to guess the future, the robot uses a "Physics Compass" (the Hamiltonian) to navigate directly toward the best solution. It's faster, more accurate, and less likely to get confused by its own imagination.

1. Problem Statement

Model-Based Reinforcement Learning (MBRL) aims to improve sample efficiency by learning a dynamics model to plan or optimize policies. However, standard MBRL approaches, particularly those using Actor-Critic architectures with Model-Based Value Expansion (MVE), face significant challenges:

Compounding Model Errors: In MVE, the critic learns from multi-step "imaginary" rollouts generated by a learned dynamics model. Errors in the dynamics model compound over the rollout horizon, leading to inaccurate value estimates.
Sensitivity to Horizon Selection: The effectiveness of MVE is highly sensitive to the choice of the rollout horizon ( $K$ ). Longer horizons introduce more bias from model errors, while shorter horizons may fail to capture long-term dependencies.
Distributional Shift: In offline RL or when initial states shift, the critic network often encounters out-of-distribution (OOD) states where value estimates become unreliable, leading to policy degradation.
Redundancy: Traditional Actor-Critic methods require learning both a policy (Actor) and a value function (Critic). The paper posits that for deterministic systems, the explicit learning of a critic network may be unnecessary if one can directly optimize based on optimal control theory.

2. Methodology: Hamiltonian Actor-Critic (HAC)

The authors propose Hamiltonian Actor-Critic (HAC), a novel MBRL algorithm that eliminates the need for an explicit critic network by leveraging the Pontryagin Maximum Principle (PMP) from optimal control theory.

Core Concept

Instead of learning a Q-value function $Q(s, a)$ , HAC reformulates the RL objective as an optimal control problem. It defines a Hamiltonian $H$ based on the learned dynamics $\hat{f}$ and the reward function. The policy is optimized by directly minimizing this Hamiltonian.

Key Technical Components

Hamiltonian Formulation:
- The RL objective is converted to minimizing a cost function $J = \sum c(s_t, a_t) + \Phi(s_T)$ .
- The Hamiltonian is defined as: $H(s_t, a_t, \lambda_{t+1}) = c(s_t, a_t) + \lambda_{t+1}^T f(s_t, a_t)$ , where $\lambda$ is the costate (adjoint variable).
- The costate $\lambda_t$ is computed via backward recursion: $\lambda_t = \nabla_{s_t} H(s_t, a_t, \lambda_{t+1})$ .
Algorithm Workflow:
- Rollout: Generate a $K$ -step imaginary rollout using the current policy $\pi_\theta$ and the learned dynamics model $\hat{f}$ .
- Backward Pass: Compute costates $\hat{\lambda}_{1:K}$ starting from the terminal condition (derived from the terminal reward) and moving backward in time.
- Policy Update: The actor loss is defined as the average Hamiltonian over the rollout: $L_{actor}(\theta) = \frac{1}{K} \sum H(\hat{s}_t, \hat{a}_t, \hat{\lambda}_{t+1})$ . The policy is updated to minimize this loss.
- No Critic Network: Since the Hamiltonian serves as the surrogate for the value function, no separate critic network is trained, removing the source of value approximation error.
Analytical Jacobian Technique:
- Computing costates requires the Jacobian of the dynamics model ( $\nabla_s \hat{f}$ ).
- Standard automatic differentiation (autograd) is inefficient for repeated backward passes through the same computational graph.
- The authors propose an Analytical Jacobian method for neural networks with ReLU activations. By exploiting the piecewise linear nature of ReLU, they derive a closed-form expression for the Jacobian using matrix multiplications and diagonal activation masks. This enables graph-free, efficient costate computation.

3. Key Contributions

Elimination of Explicit Value Learning: HAC replaces the critic network with a Hamiltonian-based formulation derived from PMP, theoretically guaranteeing convergence to a local optimum under mild smoothness assumptions.
Tighter Error Bounds: The paper proves that under specific conditions (quadratic rewards, deterministic dynamics), the maximum critic estimation error in HAC is strictly lower than that of MVE. This is attributed to the Hamiltonian Conservation Law, which prevents the accumulation of errors across the rollout horizon during the backward recursion.
Efficient Implementation: The development of the Analytical Jacobian technique allows for efficient costate computation without the overhead of repeated autograd graph traversals.
Robustness to OOD: By avoiding the training of a value function on potentially biased imaginary data, HAC demonstrates superior robustness to distributional shifts (e.g., shifted initial states) compared to MVE-based methods.

4. Experimental Results

The authors evaluated HAC on continuous control benchmarks (LQR, Pendulum, MountainCar, MuJoCo Swimmer, Hopper) in both Online and Offline settings.

Performance: HAC consistently outperformed or matched state-of-the-art baselines (DDPG, SAC, MVE-DDPG, IQL, MOPO).
- Online RL: HAC achieved faster convergence and higher final rewards, particularly in short-horizon tasks (Pendulum) and complex long-horizon tasks (Swimmer, Hopper).
- Offline RL: In data-limited settings, HAC matched or exceeded the performance of model-free offline methods (IQL, SAC-Off) and model-based methods (MOPO), demonstrating high sample efficiency.
Robustness: In OOD experiments (shifting initial states), MVE-DDPG failed to produce stable trajectories, while HAC maintained stable and consistent performance.
Critic Estimation Error: Empirical analysis on LQR tasks showed that HAC's Hamiltonian estimates were significantly closer to the ground truth than MVE's Q-value estimates, validating the theoretical error bound.
Computational Efficiency: While HAC requires backward costate computation, it avoids the need for large ensembles of critic networks (often required by MVE to mitigate errors). Ablation studies showed HAC achieves comparable performance to MVE-DDPG with a 5-critic ensemble but with lower total training time.

5. Significance

This work bridges the gap between Optimal Control Theory and Deep Reinforcement Learning.

Theoretical Insight: It demonstrates that for deterministic systems, the explicit learning of a value function is not strictly necessary if the dynamics are known (or learned), as the Hamiltonian provides a direct path to optimality.
Practical Impact: HAC offers a principled alternative to standard Actor-Critic methods that are prone to compounding errors. Its robustness to model bias and distributional shift makes it particularly suitable for offline RL and safety-critical applications where data is limited or the environment may shift.
Future Direction: The paper suggests that "model surplus" (overly complex value function approximators) may exist in current architectures, and simpler, theory-grounded approaches like HAC can yield better efficiency and stability.