Learning Sidewalk Autopilot from Multi-Scale Imitation with Corrective Behavior Expansion

1. Problem Statement

The paper addresses the challenges of sidewalk micromobility navigation (e.g., food delivery robots, assistive wheelchairs) in complex urban environments.

• Core Challenge: Current learning-based control methods, particularly Imitation Learning (IL), struggle in real-world deployment.
  • Compounding Errors: IL relies on fixed offline expert demonstrations. In closed-loop deployment, small deviations accumulate over time, leading to catastrophic failure (the "distributional shift" problem).
  • Data Limitations: Collecting data for "corner cases" (e.g., near-misses, recovery from drifting) is difficult. Standard teleoperation datasets are dominated by smooth, successful trajectories, lacking examples of how to recover from mistakes.
  • Input Complexity: The system must infer scene geometry and semantics solely from egocentric RGB videos (monocular), without pre-built maps or LiDAR.
• Goal: To develop a robust, generalizable autopilot policy that can navigate crowded sidewalks, recover from its own errors, and handle long-horizon goals using only offline teleoperation data.

2. Methodology: The MIMIC Framework

The authors propose MIMIC (Multi-scale IMItation with Corrective expan-sions), a framework that improves IL through two main pillars: Data Expansion and Multi-Scale Model Architecture.

A. Data Side: Corrective Behavior Expansion

To overcome the lack of failure-recovery data in standard teleoperation logs, the authors synthesize novel training data:

1. Trajectory Perturbation: They estimate continuous metric depth and reconstruct a 3D point cloud from the video. They then apply a deviation-recovery noise sequence (a sine-like shift) to the expert trajectory, forcing the robot to "drift" off-course (e.g., stepping onto grass or veering toward obstacles).
2. Synthesis of Failure-Correction Pairs: The system generates new observation-action pairs where the robot starts in a perturbed state and must execute a corrective action to return to the intended path. This teaches the policy how to recover from its own mistakes.
3. Sensor Augmentation: To address visual diversity (lighting, weather), they use a relighting model (Light-A-Video). This model disentangles foreground and background, applying different lighting prompts (e.g., "sunset," "snowfall") to the background while preserving foreground consistency, thereby enriching the visual distribution without breaking geometric constraints.

B. Model Side: Multi-Scale Imitation Learning

The authors introduce a novel Encoder-Decoder architecture designed to handle both immediate interactions and long-term planning.

• Multi-Scale Supervision: Instead of predicting a single trajectory, the model is supervised across four temporal horizons (Immediate, Short, Medium, Long). This prevents "shortcut learning" where the model optimizes only for immediate success at the expense of global consistency.
• Anchor-Based Query Mechanism:
  • The decoder uses K-means clustering to generate four sets of anchor trajectories corresponding to the different horizons.
  • These anchors act as structured priors. The model uses query-based attention to attend to these anchors, allowing it to capture diverse motion patterns (e.g., turning vs. straight walking) at different scales.
  • Query-Free Head: An auxiliary head predicts short-term actions directly from visual context to ensure fine-grained reactivity.
• Output: The policy outputs a Gaussian Mixture Model (GMM) over waypoints, providing a probability distribution over candidate trajectories rather than a single deterministic path.

3. Key Contributions

1. Corrective Behavior Expansion Pipeline: A method to synthesize "failure-to-recovery" trajectories from standard teleoperation data by perturbing the action-observation loop, significantly increasing the diversity and robustness of the training distribution.
2. Multi-Scale Imitation Architecture: A novel model design that utilizes horizon-specific anchor queries to jointly learn short-horizon interactive behaviors (reactivity) and long-horizon goal-directed intentions (planning) within a unified framework.
3. Real-World Validation: Successful deployment on wheeled robots in diverse US cities, demonstrating that the approach works effectively using only offline teleoperation data without requiring Reinforcement Learning (RL) or costly simulators.

4. Experimental Results

The authors evaluated MIMIC on the CoS (Coco-on-SideWalks) dataset (3,040 trajectories) and in real-world deployments.

• Open-Loop Evaluation (Offline):
  • Regular Set: MIMIC outperformed state-of-the-art baselines (GNM, ViNT, NoMaD, CityWalker) by a significant margin. It achieved a 60.6% lower minADE (Average Displacement Error) and 63.5% lower minFDE (Final Displacement Error) compared to the second-best method.
  • Recovery Set: On perturbed data (simulating drift), MIMIC showed superior robustness, reducing errors by ~50% compared to baselines, proving the effectiveness of the corrective behavior expansion.
• Closed-Loop Evaluation (Real-World):
  • Deployed on physical robots in day/night scenarios with pedestrians.
  • Success Rate: MIMIC achieved a 90% goal-reaching success rate and 76% pedestrian avoidance rate, significantly outperforming the baseline CityWalker (55% and 18%, respectively).
  • Interventions: The number of human interventions required to save the robot from failure dropped from 19 (baseline) to 4 (MIMIC) in a 400m navigation task.

5. Significance

• Bridging the Sim-to-Real Gap without RL: The paper demonstrates that high-performance, robust navigation can be achieved using purely offline data if the data distribution is intelligently expanded. This avoids the high costs and safety risks associated with Reinforcement Learning and simulator training.
• Handling Distributional Shift: By explicitly training on "recovery" scenarios, the model learns to handle the inevitable deviations that occur in real-world deployment, solving a major limitation of traditional Imitation Learning.
• Scalable Urban Autonomy: The approach provides a practical blueprint for deploying generalist sidewalk robots that can navigate complex, dynamic human environments using standard RGB cameras and GPS, without relying on expensive 3D mapping infrastructure.