Technical Summary: SidewalkBench

Problem Statement

Urban sidewalk navigation presents significant challenges for autonomous mobile robots, including delivery bots, electric wheelchairs, and e-scooters. These challenges stem from complex structural layouts, dynamic pedestrian behaviors, and the requirement for long-distance navigation. While recent visual navigation foundation models have shown promise using monocular RGB cameras, their evaluation on urban sidewalks has been hindered by a lack of unified benchmarks. Existing studies often rely on limited-scale experiments (e.g., short-horizon trials under 10 meters), lack structural diversity, and fail to isolate the impact of specific pedestrian behaviors on model performance. Furthermore, existing social navigation benchmarks primarily focus on indoor settings, leaving a critical gap in standardized, reproducible evaluation for outdoor urban environments.

Methodology

To address these gaps, the authors propose SidewalkBench, a comprehensive benchmark built upon NVIDIA Isaac Sim. The methodology is structured around three core components:

1. Simulation Platform and Scene Generation

SidewalkBench utilizes two complementary scene types to balance diversity and realism:

• Procedurally Generated Scenes: Using a pipeline inspired by MetaUrban and Urban-Sim, the authors define 7 primitive block types (straight, curves, intersections) connected via spline-based routing. These blocks are split into functional zones (roads, sidewalks, curbs, etc.) and populated with assets from the UrbanVerse-100K database. This generates 100 large-scale environments (2 km × 2 km each) to test generalization.
• Real-world Scanned Scenes: To ensure high visual and geometric fidelity, 11 real-world street blocks were scanned using a spatial camera with LiDAR and reconstructed via 3D Gaussian Splatting (3DGS). These scenes are converted into simulation-ready formats, averaging 150 m × 150 m.

2. Pedestrian Simulation

A two-level approach is employed to simulate reactive pedestrian behaviors:

• High-level Behaviors: Event-based behaviors are modeled using behavior graphs and state machines. These are dynamically triggered by the pedestrian's relative position to the robot, covering scenarios such as crossing, queuing, conversing, and gesturing.
• Low-level Animation: To overcome rendering bottlenecks and limited movement diversity in existing simulators, the authors developed a custom pipeline using the SMPL human body model. Pedestrians are rendered as textured SMPL meshes or SMPL-based 3DGS avatars, controlled by human motion generation models. This pipeline achieves a 60x improvement in rendering efficiency compared to standard Isaac Sim modules, enabling large-scale parallel evaluation.

3. Scenario Definitions

The benchmark defines three distinct testing categories:

• Unit-test Scenarios (330 total): Short-horizon tasks (10–20m) on straight, curved, and crosswalk blocks without pedestrians, focusing on lane-following and static collision avoidance.
• Pedestrian-reactive Scenarios (800 total): Tasks involving one or more pedestrians with specific event-based behaviors (e.g., obstructing, conversing, lateral crossing, gesturing) to evaluate safety and social interaction.
• Long-horizon Scenarios (105 total): Tasks requiring traversal of distances >100m (totaling 36.5 km) to mimic real-world deployment, evaluating robustness over extended periods.

Key Contributions

1. SidewalkBench Benchmark: A unified framework for standardized testing and comparison of visual navigation models on urban sidewalks.
2. Simulation Platform: A high-fidelity, GPU-accelerated environment featuring diverse procedurally generated and real-world scanned scenes, coupled with an efficient, flexible pedestrian simulation module.
3. Comprehensive Evaluation: A detailed analysis of 9 representative visual navigation models across the three scenario types, identifying specific failure modes and bottlenecks.

Experimental Results

The authors evaluated 9 models, including general foundation models (ViNT, NoMaD, MBRA, InternVLA-N1) and sidewalk-specific models (MIMIC, S2E, CityWalker, FlowPilot, OpenPilot).

• Data Scaling: Performance correlates strongly with the volume of sidewalk-specific training data. Models trained on sidewalk data (e.g., OpenPilot with 1,000 hours) significantly outperformed general foundation models.
• Pedestrian Interaction: Models struggle significantly with complex social interactions. Success rates dropped sharply in scenarios involving social groups (conversing, queuing) and specific gestures. The "Pedestrian Crossing" scenario yielded a near-zero success rate (0.01) for the average model, highlighting difficulties in handling multi-directional approaches and crosswalk rules.
• Long-horizon Robustness: Even the top-performing model (OpenPilot) experienced 1.34 failures per 100 meters in long-horizon scenarios. For a typical 2 km delivery route, this implies roughly 26.6 human interventions would be required.
• Synthetic Data Utility: A preliminary fine-tuning experiment demonstrated that training on synthetic data generated by SidewalkBench substantially improved performance in both simulation and real-world pedestrian-reactive scenarios (e.g., increasing success rates for "Ped-Crossing" from 0.11 to 0.69 in simulation).

Significance and Claims

The paper claims that SidewalkBench fills a critical void in the autonomous robotics community by providing the first dedicated benchmark for large-scale, dynamic urban sidewalk navigation. The authors assert that their findings highlight two primary bottlenecks for current technology: a lack of pedestrian behavior understanding (particularly regarding gestures and social groups) and insufficient long-horizon robustness.

The significance of the work lies in its demonstration that scaling up sidewalk training data via synthetic simulation is a promising solution to these limitations. The authors conclude that while current models show potential, substantial research is required to achieve safe, fully autonomous operation in public spaces, and SidewalkBench serves as a necessary tool to guide this development through standardized, reproducible evaluation.