Technical Summary: Hybrid Energy-Aware Reward Shaping (H-EARS)

1. Problem Statement

Deep Reinforcement Learning (DRL) for continuous control faces three persistent challenges: high variance in training, low energy efficiency, and poor generalization under distribution shifts. Purely data-driven, model-free algorithms (e.g., SAC, TD3) often rediscover fundamental physical principles through extensive trial-and-error, leading to unstable, energy-inefficient, and physically implausible policies. These policies frequently exploit simulator-specific dynamics ("shortcuts") rather than learning generalizable strategies, resulting in severe performance degradation in real-world or out-of-distribution scenarios.

Existing approaches to bridge this gap face a trade-off:

• Physics-based methods (e.g., Lagrangian or Hamiltonian Neural Networks) enforce physical consistency by learning unknown dynamics from data. However, they incur high computational complexity ($O(n^2)$ to $O(n^3)$) and require reconstructing complete system dynamics, which is often infeasible for complex systems.
• Purely model-free methods are computationally efficient but lack physical constraints, often failing when real-world dynamics deviate from training conditions.
• Standard Reward Shaping (Potential-Based Reward Shaping, PBRS) accelerates convergence but typically relies on task-specific heuristics (e.g., distance-to-goal) that lack physical interpretability and do not inherently enforce energy efficiency or stability.

The core problem is the lack of a lightweight, theoretically grounded framework that integrates known physical structure (specifically energy) into DRL to improve convergence and stability without requiring complete system identification or incurring prohibitive computational costs.

2. Methodology: H-EARS

The paper proposes Hybrid Energy-Aware Reward Shaping (H-EARS), a unified methodology that encodes dominant energy terms—assumed known a priori—directly as reward potentials. The method operates at $O(n)$ per-step computation, where $n$ is the number of generalized coordinates.

Framework Definition

H-EARS defines a shaped reward function $R_{H-EARS}$ through a two-step transformation of the Markov Decision Process (MDP):

1. Action Regularization: A new MDP $M_\lambda$ is defined with a modified reward $R_\lambda(s, a, s') = R(s, a, s') - \lambda E(a)$. Here, $E(a) = a^\top Q a$ represents the control energy functional, and $\lambda \geq 0$ is a regularization coefficient. This step intentionally modifies the optimization objective to trade task performance for energy efficiency.
2. Potential-Based Shaping: A dual-potential function $\Phi(s)$ is applied to $M_\lambda$. The final reward is:
   $$R_{H-EARS}(s, a, s') = R(s, a, s') + \gamma\Phi(s') - \Phi(s) - \lambda E(a)$$
   where $\Phi(s) = \alpha_{task}\Phi_{task}(s) + \alpha_{energy}\Phi_{energy}(s)$.
   • $\Phi_{task}(s)$: Encodes task-oriented guidance (e.g., goal proximity).
   • $\Phi_{energy}(s) = -E(q, \dot{q})$: Encodes the mechanical energy structure (kinetic + potential energy).
   • $E(a)$: Penalizes action magnitude to suppress oscillatory behaviors.

Theoretical Foundations

The paper establishes a complete theoretical framework:

• Functional Independence (Lemma II.1): The shaping term (state-dependent) and regularization term (action-dependent) act on disjoint domains. This allows $\alpha_{task}$, $\alpha_{energy}$, and $\lambda$ to be tuned independently without structurally invalidating each other.
• Gradient Enrichment (Theorem II.3): Under the condition that the energy Hessian $\frac{\partial^2 E}{\partial q^2} \succ 0$ holds over the task-relevant region, the energy potential provides informative gradient signals throughout the state space. This accelerates convergence in sparse-reward environments by ensuring $\|\nabla_q E\| \gg \|\nabla_s R\|$.
• Dual-Potential Necessity (Proposition II.4): A single potential function cannot simultaneously satisfy task directivity and energy awareness when objectives conflict. The decomposition into $\Phi_{task}$ and $\Phi_{energy}$ is mathematically necessary to balance exploration efficiency and physical plausibility.
• Convergence and Error Bounds (Theorem II.5, Lemma II.7): H-EARS achieves an $O(1/\sqrt{N})$ convergence rate under function approximation. Crucially, Lemma II.7 quantifies that approximate energy models (with up to 20% error) result in less than 5% performance loss, validating the use of simplified models.
• Stability Guidance (Proposition II.8): Maximizing the energy potential is interpreted as a Lyapunov-inspired heuristic, guiding policies toward energy-dissipating, stable behaviors.

3. Key Contributions

1. Lightweight Energy Modeling: H-EARS captures dominant energy components at $O(n)$ cost, requiring no complete dynamics derivation. This enables deployment by general engineers without analytical mechanics expertise.
2. Unified Theoretical Foundation: The paper provides formal proofs for functional independence, energy-based gradient enrichment, convergence guarantees, and approximate potential error bounds.
3. Physical Plausibility under Extreme Conditions: By embedding energy minimization as a structural prior, H-EARS constrains policies to physically consistent behaviors that maintain stability across training conditions, including extreme cases like low-adhesion roads.

4. Experimental Results

The framework was evaluated across four continuous control benchmarks (Ant-v5, Hopper-v5, LunarLander-v3, Humanoid-v5) and four baseline algorithms (SAC, TD3, PPO, DDPG), followed by high-fidelity vehicle simulations.

Standard Benchmarks

• Convergence and Stability: H-EARS consistently improved convergence speed, policy stability, and final performance. For example, SAC+H-EARS on Ant-v5 achieved a 32.5% performance gain and 28.2% faster convergence with a 27.6% reduction in variance.
• Algorithm Compatibility: The method showed the strongest benefits for stochastic off-policy methods (SAC, TD3). DDPG experienced degradation in Ant-v5 due to a conflict between its high-amplitude Ornstein-Uhlenbeck exploration and the energy regularization, highlighting a principled boundary of applicability.
• Ablation Studies:
  • Removing the task potential caused significant performance drops, confirming the need for task guidance.
  • Removing the energy potential led to instability in inherently unstable systems (Hopper-v5), validating its role in stability.
  • Removing action regularization increased variance and failed to suppress oscillatory actions, confirming its necessity for constraining action magnitude.
  • Dual-decomposition was proven superior to single-potential approaches, as merging potentials led to destructive gradient interference.

Vehicle Simulation (TruckSim)

High-fidelity simulations of a four-wheel distributed-drive electric vehicle under extreme conditions (low adhesion $\mu \in [0.1, 1.0]$, compound slopes) validated the framework's safety-critical applicability:

• Approximation Robustness: Simplified energy models with 30% approximation error still outperformed vanilla SAC in lateral stability (reducing max sideslip by 20.4% and yaw rate by 16.4%), confirming the "graceful degradation" predicted by Lemma II.7.
• Stability: H-EARS policies maintained monotonic convergence of sideslip angle and yaw rate, adhering to Lyapunov-like stability principles even under severe disturbances where vanilla SAC failed.
• Integration: The framework was successfully integrated with a lower-level Model Predictive Control (MPC) layer to enforce hard constraints, demonstrating a viable path for industrial deployment.

5. Significance and Claims

The paper claims that H-EARS offers a practical pathway from academic research to industrial deployment by demonstrating that minimal physics priors (dominant energy terms) can significantly enhance model-free RL without requiring:

• Expert-level derivation of complete system dynamics.
• Complete system identification.
• Modifications to the core RL algorithmic structures.

The significance lies in bridging the gap between the computational efficiency of model-free RL and the physical consistency of model-based methods. By providing a unified theoretical foundation that guarantees convergence, bounds approximation errors, and ensures stability, H-EARS addresses the critical challenges of modeling cost, computational complexity, and policy generalization in safety-sensitive domains. The authors emphasize that the framework is particularly effective when partial physical knowledge is available and deployment efficiency is required, complementing rather than competing with methods that learn unknown dynamics from scratch.