Enhancing Vision-Based Policies with Omni-View and Cross-Modality Knowledge Distillation for Mobile Robots

1. Problem Statement

Mobile robots relying on vision-based policies face a "trilemma" of conflicting constraints:

1. Scene Transferability: Policies trained on monocular RGB cameras often fail to generalize to unseen environments due to sensitivity to lighting, texture, and limited Field of View (FOV).
2. Onboard Computation: Lightweight robots have limited power and processing resources, making real-time processing of high-capacity encoders or large omnidirectional data streams difficult.
3. Hardware Cost: High-performance solutions often require expensive depth sensors (LiDAR, RGB-D) or multi-camera omnidirectional setups, which are unsuitable for low-cost, lightweight platforms.

The core challenge is: How can a low-cost mobile robot with a single RGB camera achieve navigation performance comparable to systems using depth sensors and omnidirectional views, without incurring high computational or hardware costs?

2. Methodology

The authors propose a Knowledge Distillation (KD) framework that transfers knowledge from a robust, information-rich "Teacher" policy to a lightweight "Student" policy.

A. Architecture Overview

• Teacher Policy ($\pi_T$):
  • Input: Omnidirectional depth images (360° view). These are generated by concatenating multi-view RGB images and converting them to depth using DepthAnyThingV2 (DATv2).
  • Strength: Provides appearance-invariant depth information and full environmental context, ensuring high safety and transferability.
  • Training: Trained via standard Imitation Learning (IL) to mimic an expert policy ($\pi_E$).
• Student Policy ($\pi_S$):
  • Input: Single-view RGB images from a standard monocular camera.
  • Architecture: Uses an image encoder (ResNet), a goal encoder, and an LSTM for temporal fusion to process image sequences.
  • Goal: To mimic the Teacher's behavior and latent representations despite having limited input data.

B. Knowledge Distillation Mechanism

The student is trained using a composite loss function ($L_S$) consisting of two components:

1. Action Loss ($L_{act}$): Standard imitation loss minimizing the $L_2$ error between the student's output actions and the expert's actions.
2. Contrastive Embedding Loss ($L_{con}$): This is the core innovation.
   • Objective: Align the latent feature embeddings ($z_2$) of the student with the embeddings ($z_1$) of the teacher.
   • Method: Uses InfoNCE contrastive loss. Positive pairs are embeddings from the same spatial state (teacher and student), while negative pairs are from distant states.
   • Effect: This forces the student to learn the "appearance-invariant" and "spatially rich" features encoded in the teacher's depth embeddings, effectively acting as a regularizer and a form of data augmentation.

C. Expert Policy Generation

The expert policy ($\pi_E$) is trained using Soft Actor-Critic (SAC) reinforcement learning based on state inputs (obstacle positions and goal coordinates), not raw images. This ensures a high-quality ground truth for the distillation process.

3. Key Contributions

1. Omni-View & Cross-Modality KD Framework: A novel method that distills knowledge from a 360° depth-based teacher to a single-view RGB student.
   • Performance Gains: Achieved a ~15% improvement in navigation success rate and a ~19% increase in collision-free travel distance compared to policies solely imitating actions.
2. Hardware and Compute Efficiency:
   • Eliminates the need for expensive depth sensors or multi-camera rigs during deployment.
   • Achieves onboard inference in ~20 ms (on an NVIDIA Jetson Orin NX), making it suitable for real-time control on lightweight robots.
3. Robustness Validation: Extensive validation in both simulation and real-world environments demonstrates that the method significantly outperforms standard pre-trained encoders (e.g., DINOv2, CLIP) and other state-of-the-art enhancement methods.

4. Experimental Results

Experiments were conducted on a lightweight mobile robot with an NVIDIA Jetson Orin NX.

• Scene Transferability:
  • Visual embedding similarity analysis showed that the proposed method maintains high consistency across different visual scenes (lighting/texture changes) compared to other RGB-based encoders.
  • t-SNE visualizations confirmed that the student's embeddings preserve meaningful spatial structures, unlike scattered embeddings from frozen pre-trained models.
• Navigation Performance (Simulation & Real World):
  • Success Rate (SR): The proposed method achieved a 72% success rate (ResNet50), significantly outperforming single-view RGB baselines (31–40%) and approaching the performance of depth-based systems (74–80%).
  • Action Error (AE): The method reduced action error to 0.0067, comparable to depth-input policies and far superior to standard RGB policies (0.0229).
  • Collision Avoidance: The robot traveled significantly further without collisions (Moving Distance: 8.49m vs. 4.08m for baseline).
• Comparison with SOTA:
  • Outperformed methods like RoboSaGA, VISARL, and Zhang et al. [12] in all metrics (AE, SR, MD).
  • Key Differentiator: Unlike methods that freeze encoders or rely solely on contrastive learning for pre-training, this method performs end-to-end training of the student policy, allowing the visual encoder to adapt specifically to the navigation task.

5. Significance

This paper addresses a critical bottleneck in mobile robotics: the trade-off between perception quality and deployment cost. By leveraging knowledge distillation, the authors demonstrate that low-cost, single-camera robots can inherit the robustness of expensive, multi-sensor systems.

• Practical Impact: The approach enables the deployment of advanced, safe navigation policies on resource-constrained robots (e.g., delivery bots, inspection drones) without requiring hardware upgrades.
• Theoretical Insight: It validates that aligning intermediate latent representations (cross-modality distillation) is more effective than simple action imitation or frozen pre-training for improving scene transferability in visuomotor control.