Technical Summary: ARCANE-PedSynth

Problem Statement

Predicting pedestrian crossing intentions in autonomous driving is a safety-critical perception task, particularly in complex urban environments. Existing real-world datasets (e.g., JAAD, PIE) suffer from severe class imbalance, where crossing events constitute fewer than 10% of observed frames, and lack sufficient representation of safety-critical edge cases such as jaywalking, sudden dashes, or mid-road pauses. Furthermore, manual annotation of subjective attributes like crossing intention is expensive and inconsistent. While synthetic data generation via simulators like CARLA offers a path to unlimited, pixel-perfect ground truth, CARLA's native AI pedestrian system is limited by its navigation mesh, which treats road surfaces as obstacles, resulting in a baseline crossing rate of only 9%. Additionally, existing synthetic solutions often lack multi-modal sensing, dense behavioral annotations beyond binary labels, or the ability to control crossing rates and simulate multi-pedestrian interactions dynamically.

Methodology

The authors present ARCANE-PedSynth, an open-source, CARLA-based software framework designed to generate synthetic multi-pedestrian datasets with dense behavioral annotations. The system is orchestrated via Docker Compose, separating the unmodified CARLA server from custom generation and pose-extraction containers to avoid dependency conflicts.

1. Hybrid AI-Manual Control Architecture

To overcome the 9% crossing rate limitation, the framework employs a dual-mode pedestrian control system:

• AI Mode: Pedestrians follow sidewalks using CARLA's native AI controllers (go_to_location).
• Manual Mode: The AI controller is stopped, and pedestrians are directly controlled via WalkerControl commands to execute specific crossing vectors, speeds, and pauses.
• Four-Layer Crossing Rate Engineering: A configurable strategy raises the effective crossing rate from 9% to 40–75% through:
  1. Crosser Allocation: Assigning 90% of spawned pedestrians as potential crossers.
  2. Behavioral Type Assignment: Distributing behaviors (50% normal, 25% sudden, 25% jaywalking).
  3. Mid-Road Injection: Spawning "mid-jaywalkers" directly in driving lanes with lateral offsets.
  4. Group Formation: Synchronizing 2–3 pedestrians into groups with a 40% probability.

2. Behavioral Finite State Machine (FSM)

Each pedestrian is driven by a 12-state FSM (e.g., Walking Sidewalk, Looking Around, Checking Traffic, Hesitating, Crossing Road, Pausing Mid-Cross, Retreat) governed by a Time-to-Collision (TTC) gap-acceptance model. Decisions to cross, pause, or retreat are based on the proximity and velocity of surrounding vehicles.

• Character Archetypes: Five distinct archetypes (business_person, casual_person, elderly_person, young_person, parent_with_child) define base parameters for speed, hesitation duration, attention levels, and intrinsic crossing propensity, informed by observational studies.
• Situational Speed: Pedestrians dynamically switch between four speed profiles (cautious, normal, rushed, distracted) to simulate real-world variability.

3. Multi-Modal Sensing and Annotation

The framework captures synchronized data from three sensors mounted on a moving ego-vehicle:

• RGB Camera: 1280×720, 90° FOV, 30 FPS.
• LiDAR: 64-channel, 100m range, 20 Hz.
• DVS (Dynamic Vision Sensor): Event camera capturing microsecond brightness changes.

Annotation Pipeline:

• Visibility Gate: Pedestrians are annotated only if within 70m, in the camera's FOV, and above a minimum bounding box size.
• Labels: Includes lane-validated binary crossing labels (C/NC), behavioral states, bounding boxes, distance to ego, and Time-to-Event (TTE) markers.
• Pose Extraction: Instead of using CARLA's internal 3D skeleton (which requires complex projection), the framework applies AlphaPose and OpenPose to the rendered RGB frames to generate COCO-17 format 2D keypoints. This approach ensures consistency with real-world processing pipelines and achieves near-perfect detection rates when supplied with ground-truth bounding boxes.

4. Reproducibility

The system is fully reproducible via CLI parameterization and Docker containerization. It supports batch generation across multiple weather conditions and towns, with a robust six-phase cleanup protocol to prevent memory leaks during extended simulation runs.

Key Contributions

1. Controllable Crossing Rates: The first open-source CARLA mechanism to configure target crossing rates up to 75%, directly addressing the class imbalance problem in real-world datasets.
2. Dense Behavioral Modeling: Introduction of a 12-state FSM with five character archetypes, enabling the simulation of nuanced behaviors (hesitation, retreat, group dynamics) absent in previous synthetic datasets.
3. Multi-Modal Synchronization: Generation of synchronized RGB, LiDAR, and DVS streams with per-frame annotations, facilitating sensor fusion research.
4. PedSynth++ Dataset: The framework was used to generate PedSynth++, a dataset comprising 533 multi-pedestrian clips across 12 weather conditions and 4 CARLA towns, containing over 177,000 annotated frames.

Results

The paper demonstrates the framework's efficacy through the PedGT model (a graph-transformer architecture):

• Performance: Trained exclusively on synthetic PedSynth++ data from Town01 and Town02, PedGT achieved an F1 score of 89 on the held-out Town05 test split.
• Temporal Windows: A 16-frame input window yielded optimal performance (F1=89), balancing context and signal dilution.
• Annotation Quality: The use of simulator-generated ground-truth bounding boxes for pose extraction resulted in significantly higher pose detection rates compared to real-world datasets (where missing joints are common), validating the internal consistency of the annotations.
• No Normalization Required: The geometrically consistent annotations preserved absolute spatial relationships (e.g., proximity to road edges), allowing models to learn effectively without aggressive data normalization.

Significance and Claims

The authors claim that ARCANE-PedSynth fills critical gaps in existing tools by providing a reproducible, open-source pipeline that combines controllable crossing rates, multi-pedestrian dynamic traffic, multi-modal sensing, and dense behavioral annotations.

• Research Enablement: The framework enables research directions previously difficult or impossible, such as training on balanced rare-event datasets, studying behavioral diversity beyond binary labels, and conducting rigorous sensor fusion experiments under varying weather conditions.
• Adversarial Training: The inclusion of rare-event states (sudden crossing, retreat, mid-road pausing) generates corner cases essential for testing autonomous vehicle robustness.
• Limitations: The authors modestly note that while the data is synthetically generated and the FSM is rule-based, it does not fully capture the richness of real-world social negotiation, gaze-based communication, or culturally dependent norms. Consequently, while the framework demonstrates strong in-domain learnability, sim-to-real generalization remains an active research direction requiring domain adaptation techniques.

The software is released under the MIT license, though third-party components for pose extraction (OpenPose, SMPL/VPoser) carry separate licensing restrictions.