Visual-RRT: Finding Paths toward Visual-Goals via Differentiable Rendering

1. Problem Statement

Traditional robot motion planning algorithms, particularly Rapidly-exploring Random Trees (RRTs), rely on explicit goal configurations defined as numerical joint angles. However, many practical robotic applications (e.g., imitation learning, human-robot interaction) specify goals via visual observations (images or videos) rather than precise joint coordinates.

Existing approaches face two main limitations when addressing visual goals:

1. Pure Sampling Methods (RRT): Cannot utilize visual goals directly because they lack a numerical target configuration to bias the search.
2. Pure Gradient Methods (Differentiable Rendering): While they can optimize robot configurations to match a goal image by minimizing rendering loss, they are prone to local minima and often fail to find a valid path in complex, cluttered environments or when the initial configuration is far from the goal.

The core challenge is to unify the global exploration capabilities of RRT with the local exploitation capabilities of differentiable rendering to plan collision-free paths toward visual goals without explicit joint targets.

2. Methodology: Visual-RRT (vRRT)

The authors propose vRRT, a motion planner that integrates sampling-based exploration with gradient-based exploitation. The method operates in the robot's configuration space (C-space) and consists of three key components:

A. Unified Tree Expansion

vRRT grows a search tree from a start configuration ($q_{start}$) by performing two parallel operations at each iteration on a batch of parent nodes:

• Random Steering (Exploration): Samples random configurations to ensure global coverage of the C-space and escape local minima.
• Visual-Gradient Steering (Exploitation): Uses differentiable robot rendering to compute the gradient of the rendering loss ($\nabla_q \mathcal{L}_{render}$) with respect to the robot's joint angles. This guides the tree expansion toward configurations that visually resemble the goal image ($I_{goal}$).

B. Frontier-Based Exploration-Exploitation Strategy

To balance the trade-off between exploring new regions and exploiting promising areas, vRRT introduces a frontier-based selection mechanism:

• Frontier Set: Maintains a set of "frontier" nodes in the current tree, ranked by their visual rendering loss (lower loss = more promising).
• Truncated Geometric Distribution: Instead of greedily selecting the best node or sampling uniformly, vRRT selects parent nodes from the frontier using a truncated geometric distribution. This adaptively prioritizes visually promising nodes while retaining a probability of selecting less promising nodes to maintain diversity.
• Anchored Sampling: Exploration steps are anchored near these selected frontier nodes, creating a "soft" bias toward the visual goal without requiring an explicit target configuration.

C. Inertial Gradient Tree Expansion

Standard gradient descent resets optimization history at every step, which is inefficient in a tree structure where branches diverge. vRRT introduces Inertial Gradient Expansion:

• State Inheritance: Each tree node maintains optimization states (first moment $m$, second moment $v$, and iteration count $i$) inherited from its parent.
• Momentum-Based Optimization: Using an Adam-like optimizer, the method accumulates momentum across the tree branches. This allows a branch to continue optimizing in a consistent direction, effectively navigating through local minima by leveraging the optimization history of its ancestors.
• Independence: Different branches maintain independent optimization trajectories, preserving the stochastic nature of RRT while enabling momentum-consistent convergence.

3. Key Contributions

1. vRRT Algorithm: The first method to unify differentiable rendering (gradient-based exploitation) with RRT (sampling-based exploration) for visual-goal motion planning, eliminating the need for explicit joint-angle goals.
2. Frontier-Based Strategy: A novel adaptive prioritization mechanism that balances goal-directed exploitation with global exploration using a rank-based probabilistic sampling distribution.
3. Inertial Gradient Expansion: A mechanism to propagate optimization states (momentum) across tree branches, enabling continuous and efficient gradient descent within the RRT framework.
4. Comprehensive Validation: Extensive experiments on three distinct robot manipulators (Franka, UR5e, Fetch) in both simulation and real-world settings.

4. Experimental Results

The authors evaluated vRRT against baselines including Dr.Robot (pure gradient), Prof.Robot (gradient with smoothing), and Dr.Robot + RRT* (two-stage approach).

• Success Rate (SR): vRRT significantly outperforms all baselines, particularly in difficult scenarios with large configuration-space distances (2.0–2.5 radians).
  • Example: On the Franka robot at a 2.5 rad distance, vRRT achieved a 44.7% success rate, whereas Dr.Robot dropped to 1.5% and Dr.Robot + RRT* to 2.0%.
• Path Quality: While pure gradient methods often fail or produce circuitous paths due to local minima, vRRT discovers paths with geometric structures similar to optimal RRT* solutions.
• Pose Reconstruction: In pose reconstruction tasks (recovering joint angles from an image), vRRT achieved near-perfect success rates (e.g., 100% on Franka for short distances) and significantly lower joint errors compared to baselines, especially under occlusion.
• Real-World Deployment: Successfully deployed on a physical Fetch robot, demonstrating effective sim-to-real transfer in cluttered tabletop environments.
• Robustness: The method remains robust even when provided with noisy goal configuration estimates, effectively fusing visual feedback with approximate kinematic hints.

5. Significance and Impact

• Bridging the Gap: vRRT bridges the divide between classical sampling-based planning and vision-centric robotics, enabling robots to plan paths based solely on "what they see" rather than "what they are told" in joint coordinates.
• Overcoming Local Minima: By combining the global search of RRT with the local precision of differentiable rendering, the method solves the local minima problem that plagues pure gradient-based visual optimization.
• General Applicability: The framework is applicable to various robot morphologies and can handle complex scenes with occlusions and clutter, making it highly relevant for real-world deployment in unstructured environments.
• Future Direction: The work opens new avenues for integrating differentiable rendering directly into sampling-based planners, suggesting potential for applications in generative robotics (planning from generated images) and human-robot interaction.

Limitations: The paper notes that visual ambiguity (e.g., symmetric poses or severe self-occlusion where different joint configurations produce identical images) remains a fundamental challenge that pure single-view planning cannot resolve without additional perceptual cues.