Hybrid TD3: Overestimation Bias Analysis and Stable Policy Optimization for Hybrid Action Space

1. Problem Statement

The paper addresses the challenges of applying Deep Reinforcement Learning (DRL) to robotic manipulation tasks that require discrete-continuous hybrid action spaces.

• The Challenge: Robotic tasks often involve high-level discrete decisions (e.g., "grasp" vs. "release") coupled with low-level continuous control (e.g., joint velocities or torques).
• Limitations of Existing Methods:
  • Discretization: Discretizing continuous components leads to exponential action space blowup.
  • Relaxation: Relaxing discrete choices into continuous approximations introduces errors and degrades policy quality.
  • Instability: Existing hybrid algorithms (e.g., Hybrid-SAC, P-DQN) often suffer from overestimation bias, particularly under full domain randomization (where object poses, masses, and scenes vary randomly). This bias creates a self-reinforcing feedback loop where noisy value estimates inflate targets, leading to training instability and poor convergence.

2. Methodology: Hybrid TD3

The authors propose Hybrid TD3, an extension of the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm, designed to natively handle parameterized hybrid action spaces.

A. Core Architecture

• Factorized Policy: The policy $\pi(a|s)$ is factorized into a discrete component $\pi_d$ (modeled as a categorical distribution via softmax) and a continuous component $\pi_c$ (deterministic actor head for joint velocities).
• Twin Critics: Two critic networks $Q_{\phi_1}$ and $Q_{\phi_2}$ jointly evaluate the state-action pair $(s, a_c, a_d)$.
• Weighted Clipped Q-Learning Target:
  • Standard TD3 selects a greedy discrete action ($\arg\max$) and applies a clipped minimum over two critics.
  • Hybrid TD3 Innovation: Instead of committing to a single greedy discrete action, the method marginalizes the target over the full discrete action distribution.
  • Target Equation:
    $$y = r + \gamma \frac{1}{K} \sum_{k=1}^{K} \pi_d(a_d^{(k)} | s') \min_{i=1,2} Q'_{\phi_i}(s', a'_c, a_d^{(k)})$$
  • This approach propagates a "soft," distribution-weighted value estimate, smoothing gradient signals and reducing sensitivity to discrete misclassification during early training.

B. Theoretical Analysis of Overestimation Bias

The paper provides a rigorous theoretical analysis establishing a formal ordering of expected estimation bias ($\delta_Q$) across five hybrid algorithmic variants.

• Key Finding: The authors prove that Hybrid TD3 achieves the lowest expected bias (most negative/underestimation), followed by HyACC/HyTQC, HyDARC, and finally HyDATD3 (least negative/most prone to overestimation).
• Implication: In highly stochastic environments (full domain randomization), a tightly bounded underestimation (pessimism) is more stable and beneficial than attempting to eliminate bias entirely or allowing overestimation. The "clipped double Q-learning" mechanism effectively controls this bias.

3. Key Contributions

1. Systematic Empirical Analysis: Identified TD3 as the most stable backbone for hybrid action RL under full domain randomization, outperforming SAC, DDPG, and PPO due to its deterministic policy and twin-critic architecture.
2. Weighted Clipped Q-Learning Target: Introduced a novel target function that marginalizes over the discrete action distribution. This improves gradient smoothness while preserving the bias-reduction properties of standard TD3.
3. Rigorous Bias Ordering: Derived formal bounds and a complete ordering of estimation bias for five hybrid variants, providing principled guidance for algorithm selection.
4. Zero-Shot Generalization: Demonstrated that the proposed method generalizes effectively to novel object categories unseen during training.

4. Experimental Results

The method was evaluated on a UF850 robotic arm in simulation (PyBullet) across four manipulation tasks: Reach, Pick, Move, and Put.

• Comparison with Bias-Mitigation Variants:
  • Hybrid TD3 achieved the highest average return and fastest convergence compared to HyACC, HyTQC, HyDARC, and HyDATD3.
  • While HyACC and HyTQC showed high positive bias (overestimation) due to their stochastic nature, Hybrid TD3 maintained tightly bounded, near-zero or slightly negative bias, leading to superior stability.
• Comparison with Dedicated Hybrid Baselines:
  • Hybrid TD3 outperformed state-of-the-art hybrid algorithms (Hybrid-SAC, TD3AQ, TSMPDQN, PASAC, ADSAC).
  • Baselines like TSMPDQN and PASAC exhibited volatile training curves and frequent collapse, whereas Hybrid TD3 showed consistent learning.
• Generalization:
  • The policy achieved high success rates on unseen object sets (different shapes, sizes, textures) without retraining.
  • Success Rates (Unseen Set): Reach (94.25%), Pick (90.00%), Move (81.75%), Put (82.75%).
  • The minimal performance drop between standard and unseen sets confirms robust zero-shot generalization.

5. Significance

• Robustness in Real-World Scenarios: The paper demonstrates that for robotic manipulation under heavy uncertainty (domain randomization), controlled underestimation (via clipped double Q-learning) is superior to attempts at unbiased or over-optimistic estimation.
• Algorithmic Guidance: The theoretical bias ordering provides a clear framework for selecting algorithms in hybrid settings, suggesting that deterministic, twin-critic approaches are preferable to stochastic or distributional methods in high-variance environments.
• Practical Applicability: By successfully handling the complexity of discrete mode selection and continuous control simultaneously, Hybrid TD3 offers a viable path toward deploying robust, adaptive robotic agents in unstructured environments.

In conclusion, Hybrid TD3 resolves the instability of hybrid action RL by leveraging the structural advantages of TD3 (twin critics, delayed updates) and enhancing them with a distribution-marginalized target, resulting in a state-of-the-art solution for complex robotic manipulation tasks.