3D RL-DWA: A Hybrid Reinforcement Learning and Dynamic Window Approach for Goal-Directed Local Navigation in Multi-DoF Robots

Technical Summary: 3D RL-DWA for High-DoF Robot Navigation

Problem Statement
Autonomous navigation in complex, constrained 3D environments remains a significant challenge for high-degree-of-freedom (DoF) robotic systems, particularly when operating under sparse sensory conditions. Traditional global planning methods often fail in dynamic, partially known environments, while standard local planners like the Dynamic Window Approach (DWA) rely on fixed parameter tuning that limits adaptability. Furthermore, existing hybrid Reinforcement Learning (RL) and DWA approaches have predominantly focused on 2D motion in structured settings, leaving a gap in solutions for 3D control of deformable, high-DoF robots (such as microrobots) that must simultaneously optimize motion and shape to navigate tortuous structures like vascular networks.

Methodology
The authors propose 3D RL-DWA, a hybrid framework that integrates Reinforcement Learning (RL) with a customized 3D Dynamic Window Approach (DWA) for the local navigation of a 9-DoF deformable robot. The system is modeled as an ellipsoid with 6 DoF for pose (position and orientation) and 3 DoF for shape deformation along its symmetry axes.

• Control Architecture: The framework operates in a closed loop where an RL agent observes the environment (task state and sparse laser scan data) and outputs two types of commands:

  1. DWA Parameters: The agent dynamically adjusts the weighting factors ($\alpha, \beta, \gamma, \zeta$) of the DWA cost function, which balances speed, direction, obstacle clearance, and goal alignment.
  2. Velocity Commands: The agent directly outputs angular velocity ($\omega_x$) and deformation velocities ($\dot{\delta}$).
     The DWA module then utilizes these parameters to select the optimal linear velocity from an admissible set, ensuring collision-free trajectories within a 3D dynamic window.

• Reinforcement Learning Formulation: The problem is formulated as a Markov Decision Process (MDP) using the Soft Actor-Critic (SAC) algorithm. The agent is trained to maximize a dense reward function that encourages forward progress, collision avoidance, and the maximization of the robot's occupied volume within the vessel. The observation space includes the robot's deformation state, normalized velocity, displacement vector to the goal, and $N$ laser scan distances.

• Kinematics: The robot's motion is governed by a simplified kinematic model inspired by ferrofluidic systems. A key constraint is that the robot's x-axis is aligned with the direction of motion, reducing the controllable rotational DoF to a single axis ($\omega_x$), while the remaining rotational components are passively induced by alignment constraints.

Key Contributions

1. Novel Hybrid Framework: The paper introduces the first application of an RL-DWA hybrid specifically for 3D local navigation of high-DoF deformable robots, addressing the gap in 3D motion control for such systems.
2. Adaptive Parameter Tuning: Unlike traditional DWA which requires manual tuning, the proposed method uses RL to dynamically adjust DWA cost function weights in real-time based on environmental conditions and sensor sparsity.
3. Simultaneous Motion and Shape Control: The framework co-optimizes pose and shape deformation, enabling the robot to maximize its volume occupation while navigating constrained spaces.
4. Robustness to Sparse Sensing: The system is explicitly evaluated under varying levels of exteroceptive data (15, 25, and 50 sensor points), demonstrating adaptability in conditions typical of biomedical applications where high-resolution sensing is limited.

Experimental Results
The framework was evaluated in a simulated vascular network environment using 1,080 experimental trials across training and unseen test scenarios.

• Performance Comparison: The 3D RL-DWA significantly outperformed both a pure RL baseline (SAC) and a model-based controller (using Delaunay triangulation and Voronoi graphs).
  • Path Completion: In unseen test scenarios with high-resolution sensing ($N=50$), 3D RL-DWA achieved near-perfect path completion (median ~99.22%), whereas the pure RL baseline struggled (median ~45.75%) and the model-based method failed entirely at lower resolutions.
  • Deformation: The hybrid approach maintained high deformation capabilities (median ~53-57% volume occupation) across resolutions, while pure RL showed degraded deformation performance at lower sensor densities due to navigation failures.
• Sensor Resolution Impact: Higher sensor resolution ($N=50$) improved convergence speed and reproducibility. However, the system demonstrated robustness even at intermediate resolutions ($N=25$), while $N=15$ resulted in reduced navigation reliability but preserved some deformation capability.
• Computational Efficiency: The system demonstrated real-time suitability with mean inference times consistently below 2 ms per control step. The RL component was constant in time, while the DWA component scaled linearly with sensor points but remained efficient.
• Robustness to Noise: Under increasing Gaussian noise, the 3D RL-DWA maintained superior path completion rates compared to baselines, though performance naturally degraded with noise. Deformation control proved inherently more robust to sensor noise than navigation.

Significance and Claims
The authors claim that the 3D RL-DWA framework effectively bridges the gap between the adaptability of learning-based methods and the structured guidance of classical planners. The hybrid approach is presented as a solution that achieves a balance neither pure learning nor purely model-based methods can provide in constrained 3D environments.

The paper emphasizes that this method is particularly effective for emerging biomedical applications, such as the autonomous control of microrobots for minimally invasive interventions, where robots must navigate unstructured, sparse-sensing environments. The results suggest that integrating RL to tune DWA parameters allows for robust, adaptive navigation that generalizes well to unseen waypoint sequences and sensor uncertainties, making it a viable candidate for real-time deployment in complex 3D spaces.