Path-conditioned Reinforcement Learning-based Local Planning for Long-Range Navigation

1. Problem Statement

Long-range navigation in autonomous robots is typically handled via hierarchical pipelines: a global planner generates a coarse path (waypoints), and a local planner executes short-range motions. Current systems face two primary challenges:

• Sensitivity to Global Path Quality: If the global planner produces inaccurate paths due to poor remote sensing (e.g., noisy maps), local planners often fail to execute them, leading to infeasible trajectories.
• Limited Global Context: Local planners often lack sufficient global context (beyond a single waypoint), forcing them to rely on reactive exploration. This leads to inefficient trajectories, repeated dead-end traversals, and poor long-range performance.

Existing solutions either enforce strict path following (limiting adaptability to obstacles) or ignore path information entirely (leading to inefficient exploration). The authors propose a middle ground: a local planner that opportunistically uses global path information as guidance without treating it as a rigid constraint.

2. Methodology

The authors propose a Path-Aware Reinforcement Learning (RL) framework that conditions a local navigation policy on reference path observations.

A. Problem Formulation

The task is modeled as a Partially Observable Markov Decision Process (POMDP). The agent operates in an unknown 3D environment using only a front-facing depth camera and proprioceptive sensors to reach a goal. The state is estimated by fusing current observations with a history of past observations.

B. Path Representation and Encoding

To provide global context without enforcing strict adherence, the entire reference path is encoded as a sequence of $N$ waypoints relative to the robot's egocentric frame.

• Augmented Encoding: Each waypoint is transformed into a 4D vector containing normalized direction, log-scaled distance (to handle large variations), and a normalization constant.
• Attention Mechanism: The policy utilizes a Self-Attention mechanism to capture global path structure and a Cross-Attention mechanism with a learnable query. This allows the policy to selectively focus on specific, informative segments of the path (e.g., sharp turns or critical deviations) relevant to the current context.
• Fusion: The resulting path embedding is concatenated with visual features from a depth encoder and proprioceptive data within a Spatially-Enhanced Recurrent Unit (SRU).

C. Training Strategy

The training process is designed to teach the agent to distinguish between useful and misleading guidance:

• Path Generation: Reference paths are sampled from a distribution including optimal paths (A*) and suboptimal paths (Greedy Best-First Search with biased heuristics).
• Noise Injection: Waypoints are perturbed with noise (up to 2m displacement) to simulate sensor errors or map inaccuracies, making some paths locally infeasible.
• Reward Design: The reward function is purely goal-driven (task completion, smoothness, safety penalties). Crucially, there is no explicit path-following reward.
• Shortcut Reward: A specific reward term ($r_{shortcut}$) is introduced to encourage the agent to bypass unnecessary detours if it detects a more direct route than the reference path suggests. This reinforces "opportunistic deviation."

D. Architecture

The system uses an Asymmetric Actor-Critic PPO algorithm. The policy is trained end-to-end, jointly learning state estimation and action selection.

3. Key Contributions

1. Path-Conditioned Architecture: A novel RL policy that fuses structured global path representations with local perceptual observations using attention mechanisms, enabling the agent to reason about long-horizon structures.
2. Opportunistic Training Strategy: A training regime that exposes the agent to noisy and suboptimal paths without explicit path-following rewards. This teaches the policy to treat path information as contextual guidance rather than a constraint, allowing it to ignore misleading cues.
3. Real-World Validation: Successful deployment on a Unitree B2W quadruped robot in complex indoor environments, demonstrating robust sim-to-real transfer.

4. Experimental Results

The method was evaluated in simulation (Isaac Sim) and real-world scenarios across procedurally generated terrains.

• Performance with High-Quality Paths: When provided with optimal reference paths, the proposed model achieved a 7.02% increase in Success weighted by Path Length (SPL) compared to a baseline RL policy, demonstrating efficient exploitation of global guidance.
• Robustness to Degraded Paths: When paths were noisy or suboptimal, the model's performance remained comparable to the baseline (SPL ~0.74 vs 0.75). Unlike rigid path-following agents, it did not degrade significantly when the guidance was misleading.
• Ablation Studies:
  • Architecture: The variant using learned cross-attention queries outperformed fixed queries and direct concatenation, indicating the importance of selective focus.
  • Training: Models trained without waypoint noise failed to generalize to degraded paths, confirming that exposure to imperfect data is crucial for robustness.
  • Missing Inputs: The agent could navigate effectively using only visual inputs (when path was removed) or only path inputs (when vision was removed), though visual inputs were superior for complex terrain.
• Real-World Deployment: In a university building, the robot successfully navigated ~90m trajectories. It used the path to bias exploration toward safer routes (avoiding stairs) even when the provided path was imperfect, and successfully deviated when the path led to dead ends.

5. Significance

This work bridges the gap between purely exploratory RL navigation and rigid path-following control. Its significance lies in:

• Adaptability: It solves the "brittleness" of hierarchical planners. If the global plan is wrong, the local planner can ignore it; if the plan is good, it leverages it for efficiency.
• Scalability: It enables long-range navigation in environments where high-level maps are approximate or non-existent, a common scenario in real-world robotics.
• Implicit Reliability Estimation: The system learns to implicitly gauge the reliability of path information through training distribution rather than requiring explicit uncertainty estimates, making it a practical solution for deployment in uncertain environments.

In conclusion, the proposed method creates a local planner that is robust to uncertainty and efficient when information is available, making it highly suitable for long-range autonomous navigation in dynamic and imperfectly mapped environments.