🤖 AI

Proximal Action Replacement for Behavior Cloning Actor-Critic in Offline Reinforcement Learning

This paper introduces Proximal Action Replacement (PAR), a plug-and-play training sample replacer that overcomes the performance ceiling of behavior cloning-regularized actor-critic methods in offline reinforcement learning by substituting suboptimal dataset actions with improved ones guided by value function ascent, thereby consistently enhancing performance across benchmarks.

Original authors: Jinzong Dong, Wei Huang, Jianshu Zhang, Zhuo Chen, Xinzhe Yuan, Qinying Gu, Zhaohui Jiang, Nanyang Ye

Published 2026-05-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jinzong Dong, Wei Huang, Jianshu Zhang, Zhuo Chen, Xinzhe Yuan, Qinying Gu, Zhaohui Jiang, Nanyang Ye

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to walk using a video library of someone else's walking attempts. This is the world of Offline Reinforcement Learning (RL). The robot can't try things out in the real world (that's too dangerous or expensive); it can only learn from a static dataset of past actions.

The Problem: The "Bad Teacher" Trap

Most current methods use a technique called Behavior Cloning (BC). Think of this as the robot trying to perfectly mimic the teacher in the video.

The Good: It keeps the robot safe and prevents it from trying crazy, dangerous moves it hasn't seen before.
The Bad: What if the teacher in the video was actually a clumsy walker who kept stumbling? If the robot blindly copies every move, it will learn to stumble too. Even if the robot's "brain" (the Critic) realizes, "Hey, lifting the foot higher would be better," the "mimicry rule" (BC) forces it to keep copying the clumsy foot-dragging because that's what the data says.

The paper argues that this blind imitation creates a performance ceiling. The robot gets stuck in a "good enough" zone and can never reach "great" because it's too afraid to stop copying the suboptimal data.

The Solution: "Proximal Action Replacement" (PAR)

The authors propose a new method called PAR. Instead of just copying the teacher, PAR acts like a smart editor who watches the video and swaps out the bad moves for better ones before the robot learns them.

Here is how PAR works, using two creative metaphors:

1. The Compass (Gradient-Direction-Aware Action Replacement)

Imagine the robot is standing on a hill, and the goal is to reach the very top (the best possible reward).

The Old Way: The robot looks at the video and sees the teacher walking in a circle. It copies the circle.
The PAR Way: The robot has a "Compass" (the Critic network) that points toward the steepest path up the hill.
- It looks at the teacher's move. Does it point toward the top?
- It looks at a "Target Policy" (a slightly smarter version of the robot) and asks, "If you moved, where would you go?"
- The Swap: If the Target Policy's move points more directly toward the top of the hill than the teacher's move does, PAR replaces the teacher's bad move with the Target's good move in the training data. It's like editing the video to show the teacher taking the correct step instead of the wrong one.

2. The Seatbelt (Gaussian-Shaped OOD Action Weighting)

There is a risk here. If the robot starts suggesting moves that are too different from the original video, the "Compass" might get confused and give bad advice (this is called the "Out-of-Distribution" problem).

The Solution: PAR wears a "Seatbelt." It measures how far a new, suggested move is from the original data's comfort zone.
If the new move is a little bit better but still safe, the Seatbelt is loose, and the robot learns from it.
If the new move is wild and far away from anything seen in the data, the Seatbelt tightens (the weight drops to near zero), and the robot ignores that move to prevent crashing. This ensures the robot improves without going off the rails.

The Results: Breaking the Ceiling

The paper tested this on various robotic tasks (like walking, running, and manipulating objects) using standard benchmarks.

The Outcome: By swapping out the bad moves for better ones while keeping the training stable, PAR consistently helped robots perform better than before.
The Comparison: In many cases, simply adding PAR to a basic, simple algorithm (TD3+BC) made it perform as well as, or even better than, much more complex, state-of-the-art algorithms.

Summary

Think of PAR as a smart coach who watches a student's practice video. Instead of just telling the student, "Copy exactly what you see," the coach says, "I see you trying to do X, but looking at the physics of the situation, doing Y would be better. Let's practice Y instead, but only if Y is safe and close to what we know works."

This allows the student to break free from the limitations of the original, imperfect data and learn to perform at a truly optimal level.

Technical Summary: Proximal Action Replacement for Behavior Cloning Actor-Critic in Offline Reinforcement Learning

1. Problem Statement

Offline Reinforcement Learning (RL) aims to learn optimal policies from static, previously collected datasets without further environment interaction. A dominant approach to ensuring stability in offline RL is Behavior Cloning (BC) regularization, which constrains the learned policy to remain close to the behavior policy present in the dataset. This mitigates the distribution shift problem and prevents the agent from exploiting out-of-distribution (OOD) actions.

However, the authors identify a fundamental limitation: BC regularization imposes a performance ceiling. When the offline dataset contains suboptimal actions, indiscriminate imitation forces the actor to converge toward these suboptimal regions, even when the critic (value function) identifies better actions. The authors argue that this creates a structural trade-off where the policy cannot fully exploit the value function's suggestions for improvement, particularly in later training stages when the primary bottleneck shifts from "realism" to "optimality."

2. Methodology: Proximal Action Replacement (PAR)

To break this performance ceiling while maintaining training stability, the authors propose Proximal Action Replacement (PAR), a plug-and-play training sample replacer. PAR dynamically modifies the supervision targets for the actor's behavior cloning loss. Instead of blindly imitating the logged dataset action, PAR selectively replaces it with a better action generated by a stable target policy, subject to two key constraints:

A. Gradient-Direction-Aware Action Replacement (GDAAR)
This component determines when to replace an action.

Mechanism: For a given state $s$ , the method compares the logged action $a$ and a candidate action $\hat{a}$ (generated by a target actor $\pi'$ ) against the local ascent direction of the critic's action-value function $Q(s, a)$ .
Metric: It calculates the cosine similarity between the displacement vectors ( $v = a - \pi_\theta(s)$ and $v' = \hat{a} - \pi_\theta(s)$ ) and the critic's gradient $g = \nabla_a Q(s, a)|_{a=\pi_\theta(s)}$ .
Decision Rule: If the target action $\hat{a}$ is more aligned with the critic's local ascent direction than the logged action $a$ (i.e., $\cos(g, v') > \cos(g, v)$ ), the BC supervision target is replaced with $\hat{a}$ . Otherwise, the original action $a$ is retained.
Theoretical Guarantee: The authors prove that under smoothness assumptions, this selection rule ensures the one-step value improvement is no worse than vanilla BC up to second-order error.

B. Gaussian-Shaped OOD Action Weighting (GSOAW)
This component determines how much to trust the replacement, ensuring stability against distributional shift.

Motivation: Replacing actions can cause the policy to drift significantly from the behavior policy, leading to critic collapse (instability) due to extrapolation errors.
Mechanism: PAR introduces an auxiliary state-value function $V(s)$ and learns a standard deviation $\sigma(s)$ to model the uncertainty of $Q(s, a)$ with respect to actions.
Weighting: A weight $w$ is assigned to the replacement action based on the deviation of its Q-value from the expected value $V(s)$ , normalized by the uncertainty $\sigma(s)$ :
$w = \exp\left(-\frac{|Q(s, \hat{a}) - V(s)|}{\tau \sigma(s) + \epsilon}\right)$
Effect: If the generated action is far from the distribution of values seen in the dataset (high uncertainty or large deviation), $w$ approaches 0, effectively suppressing the replacement and reverting to the conservative behavior anchor. This adaptively bounds policy divergence without hard constraints.

3. Key Contributions

Theoretical Characterization of Suboptimality: The authors formally prove (Theorem 3.1) that for BC-regularized objectives (including MSE, KL, and MLE forms), if the dataset contains suboptimal actions, the stationary point of the optimization cannot coincide with the true optimal policy. The policy is structurally forced to compromise between imitation and optimality.
Proximal Action Replacement (PAR): A novel, plug-and-play module that integrates with existing BC-regularized actor-critic algorithms. It combines GDAAR to guide policy improvement toward higher-value regions and GSOAW to maintain stability by penalizing risky OOD actions.
Extensive Empirical Validation: The method is tested across diverse offline RL benchmarks (MuJoCo, AntMaze, FrankaKitchen) and integrated with multiple baseline algorithms (BCQ, TD3+BC, IQL, EDP, SSAR).

4. Experimental Results

Performance Gains: PAR consistently improves performance across all tested algorithms and domains.
- On MuJoCo tasks, combining PAR with TD3+BC increased the average normalized score from 75.3 to 87.7 (+12.4).
- On AntMaze, PAR enabled IQL to improve from 57.1 to 70.8 (+13.7) and rescued TD3+BC from near-zero performance (0.2 to 2.0).
- On FrankaKitchen, PAR improved TD3+BC from 0.0 to 20.0 and IQL from 48.7 to 61.5.
State-of-the-Art Comparison: The combination of SSAR + PAR achieved the highest average score (90.9) on MuJoCo tasks, outperforming complex state-of-the-art methods like QIPO (87.4) and other trajectory planning approaches.
Ablation Studies:
- Removing the action replacement (Filtering only) yielded smaller gains, confirming that introducing better actions is crucial.
- Removing GSOAW ("No GSOAW") caused severe performance degradation on medium-expert datasets due to OOD instability, highlighting the necessity of the uncertainty-aware weighting.
Efficiency: PAR introduces minimal computational overhead (approx. 3-8% increase in training time) as it reuses existing gradient computations and adds only lightweight vector operations.

5. Significance and Claims

The paper claims that PAR offers a new, data-centric perspective on overcoming the structural limitations of offline RL. By shifting the paradigm from pure imitation to guided policy improvement, PAR allows agents to transcend the performance ceiling imposed by standard behavior cloning.

The authors emphasize that PAR is modest in its scope:

It is currently designed for continuous action spaces where local ascent gradients can be computed.
It operates at the single-step transition level and does not inherently capture long-horizon dependencies (though this is noted as future work).
It is presented as a plug-and-play module that enhances existing algorithms rather than replacing the entire offline RL framework.

Ultimately, the work demonstrates that by dynamically optimizing the supervision signal itself—replacing suboptimal data points with better, uncertainty-bounded alternatives—offline RL algorithms can achieve significantly higher performance without sacrificing the stability provided by behavior cloning.