Learning Category-level Last-meter Navigation from RGB Demonstrations of a Single-instance

Technical Summary: Learning Category-level Last-meter Navigation from RGB Demonstrations of a Single-instance

Problem Definition
The paper addresses a critical gap in mobile manipulation: the mismatch between global navigation and manipulation execution. While global navigation systems typically guarantee coarse, meter-level accuracy (e.g., stopping within 1 meter of a target), manipulation policies require centimeter-level precision in position and degree-level precision in orientation to operate reliably within their training distribution. Existing solutions for high-precision positioning often rely on expensive or brittle priors, such as high-resolution maps, neural radiance fields (NeRFs), depth cameras, or LiDAR. The authors define the "last-meter navigation" problem as the specific stage between global path planning and manipulation, aiming to achieve precise alignment relative to a target object using only onboard RGB observations, without depth sensors, LiDAR, or map priors.

Methodology
The authors propose an object-centric imitation learning framework trained on a single instance of an object category (e.g., a specific green chair) that generalizes to unseen instances. The system is formulated as a behavioral cloning (BC) task where a policy $\pi$ maps multi-modal inputs to discrete motion primitives (forward, lateral, rotation).

1. Inputs: The policy conditions on three inputs:

   • Current Observation ($O_t$): Multi-view RGB images (front, back, left, right) from onboard cameras.
   • Goal Observation ($O_{goal}$): RGB images captured at the desired terminal pose.
   • Text Prompt ($\phi_{text}$): A natural language description of the target object.

2. Architecture: The system consists of three modules (Fig. 2):

   • Vision Encoder: Uses a frozen, pretrained DINOv2 encoder to extract semantically rich feature embeddings from current and goal images.
   • Segmentation Module: A two-stage text-driven process using OwlV2 for initial object detection and bounding box generation, followed by SAM2 to refine the segmentation mask. This module isolates the target object from the background.
   • Action Decoder: The decoder integrates the bounding box coordinates (encoded as a box embedding) with the masked visual embeddings. Crucially, it computes a spatial score matrix ($S \in \mathbb{R}^{64 \times 64}$) representing the pairwise cosine similarity between spatial tokens of the current and goal views. This matrix explicitly encodes the spatial offset between the current view and the goal. The flattened matrix and box embedding are fed into a feedforward network to predict discrete actions.

3. Auxiliary Stopping Mechanism: To address the difficulty of learning precise termination from noisy demonstration data, the authors introduce a rule-based override. The robot stops if the Intersection over Union (IoU) between the current and goal masks exceeds 0.6 and the horizontal center-of-mass difference is less than 25 pixels.

Key Contributions

1. Object-Centric Framework: A novel imitation learning framework that solves the last-meter navigation problem, producing manipulation-ready base poses using only RGB inputs.
2. Instance-to-Category Generalization: Demonstration that a model trained on a single object instance can reliably transfer to unseen instances of the same category across diverse environments and lighting conditions.
3. Sensor-Efficient Validation: Real-world validation proving that precise last-meter navigation is achievable without depth, LiDAR, or map priors, addressing practical constraints where such sensors may be unavailable or cost-prohibitive.

Experimental Results
The system was evaluated on a Boston Dynamics Spot robot in indoor and outdoor environments.

• Metrics: Two metrics were introduced: Edge Alignment (evaluating alignment with a ground-truth linear boundary) and Object Alignment (evaluating visual facing of the target object via center-of-mass distance).
• Performance:
  • On the seen instance (training chair), the system achieved 94% success in ground-truth evaluation and 96.67% in visual evaluation.
  • On unseen instances (different chairs), the system achieved 74.58% success in edge alignment and 89.42% in object alignment.
  • The framework also generalized to other categories (drawer organizers and suitcases) with average success rates of 60–96% depending on the metric and category.
• Ablation: Comparisons showed that implicit visual grounding (without explicit segmentation) failed completely (0% success), and the explicit Score Matrix was critical for category-level generalization compared to standard cross-attention mechanisms. The auxiliary stopping mechanism was essential for converting navigational success into task completion.
• Failure Modes: The primary failure modes were attributed to segmentation instability in challenging lighting conditions (e.g., dim environments) and the inherent noise in demonstration data regarding final stop positions.

Significance and Claims
The paper claims that precise last-meter navigation can be achieved at a category level using only RGB observations, offering a scalable pathway toward unified mobile manipulation. By removing the dependency on depth sensors, LiDAR, or pre-built maps, the approach lowers the hardware barrier for deploying autonomous mobile manipulators in dynamic, real-world human environments. The authors emphasize that their method bridges the "last-meter" gap, enabling robots to reach manipulation-ready poses that allow downstream manipulation policies to operate within their training distribution, thereby reducing execution failures. The work suggests that while current performance is limited by segmentation quality and unobstructed visibility, the proposed object-centric, data-driven approach provides a robust foundation for future advancements in vision-based mobile manipulation.