Multi-View In-Cabin Monitoring System for Public Transport Vehicles

Technical Summary: Multi-View In-Cabin Monitoring System for Public Transport Vehicles

Problem Statement
While autonomous driving has advanced significantly in external perception, understanding the interior of vehicles—specifically in public transportation—remains underdeveloped. In-cabin monitoring is critical for occupant counting, localization, accessibility assistance, and safety (e.g., detecting aggression or vandalism). However, existing solutions face a data bottleneck: current in-cabin datasets are often limited in scale, rely on single-camera views, and lack the multi-modal synchronization required to handle severe occlusions, crowding, and reflective surfaces common in bus interiors. Furthermore, multi-sensor calibration is challenging in buses because inward-facing cameras have minimal field-of-view overlap, making standard calibration techniques difficult to apply.

Methodology
The authors present a comprehensive pipeline for capturing, calibrating, and annotating a multi-view in-cabin dataset from a digitized German city bus.

1. Data Acquisition:

   • Sensors: The setup utilizes four inward-facing Intel RealSense D435i cameras (providing RGB and stereo depth) and a rotating Ouster OS0-128 LiDAR.
   • Synchronization: Cameras run at 15 Hz and LiDAR at 10 Hz. Data is synchronized via timestamps, pairing each LiDAR frame with the temporally closest camera frames.
   • Content: The dataset captures 1–2 occupants performing various activities (sitting, standing, walking, drinking, smoking) and safety-critical events (vandalism, fighting, wheelchair entry).

2. Sensor Calibration:

   • Extrinsics: Due to limited camera overlap, the authors employ a two-stage calibration process. First, they reconstruct the cabin geometry using COLMAP (Structure from Motion and Multi-View Stereo) based on scattered ArUco markers.
   • Metric Alignment: The reconstruction is scaled and oriented using a known marker size and a specific "origin marker" placed at the vehicle's centerline to establish a ground-aligned metric frame.
   • LiDAR Alignment: The LiDAR point cloud is registered to the COLMAP reconstruction using Iterative Closest Point (ICP), achieving a Root Mean Square Error (RMSE) of 0.0071 m.

3. Pseudo-Labeling Pipeline:

   • 2D Detection: A 2D detector identifies candidate persons, followed by manual filtering to remove false positives (e.g., posters).
   • 3D Pose Estimation: The pipeline uses SAM 3 Body 3D to generate 3D human meshes and skeletons from the 2D detections and known camera intrinsics.
   • Multi-View Aggregation: To resolve discrepancies between views (which can differ by up to ~1 m), the authors implement a three-step aggregation:
     1. Camera-constrained clustering: Clusters pose hypotheses while ensuring no two detections from the same camera enter the same cluster.
     2. Median pose selection: Selects the hypothesis closest to the cluster centroid to reduce outlier impact.
     3. Refinement: Optimizes the global translation of the selected pose to minimize multi-view reprojection error using the Nelder–Mead algorithm.
   • 3D Bounding Boxes: Oriented 3D bounding boxes are derived from the refined 3D meshes, defined by center, dimensions, and yaw angle relative to the subject's hip rotation.

4. Dataset Format:

   • The final dataset is converted to the nuScenes format, facilitating compatibility with existing autonomous driving perception codebases.

Key Contributions

• Dataset: A multi-view, multi-modal (RGB, depth, LiDAR) in-cabin dataset containing 9,136 synchronized samples with 3D human pose and oriented 3D bounding box annotations. It is the first publicly available dataset of this kind with 3D pose annotations.
• Pipeline: A practical calibration and pseudo-labeling pipeline capable of generating high-quality 3D annotations in a challenging, low-overlap multi-view environment.
• Benchmark: Baseline results for state-of-the-art multi-view 3D detection models (Lift-Splat-Shoot, BEVFusion) on in-cabin data.

Experimental Results
The authors evaluated several models on a 7,128/2,008 train/validation split:

• Camera-Only Models: Lift-Splat-Shoot (LSS) with ResNet-50 and SwinT backbones achieved strong performance, with Average Precision (AP) at 0.5m reaching 0.9630 (ResNet) and 0.7563 (SwinT).
• Multi-Modal Models: BEVFusion (fusing LiDAR and cameras) performed worse (AP@0.5m: 0.7337) than the camera-only LSS variants. The authors attribute this to the model's heavy reliance on LiDAR, which is less effective in the constrained, sparse interior environment compared to the dense camera features.
• View Analysis: Single-view detection miss rates ranged from 18.78% (Front Left) to 60.69% (Center Left), demonstrating that multi-view fusion is essential for robust in-cabin perception.

Significance and Claims
The paper positions this work as a foundational step toward robust in-cabin perception for public transport. The authors claim that:

1. Data Scarcity is Addressed: The dataset fills a critical gap by providing a large-scale, multi-modal resource specifically for the challenging geometry of vehicle interiors.
2. Calibration Feasibility: The proposed calibration pipeline successfully handles the low-overlap constraints of inward-facing cameras, enabling metrically consistent 3D reconstruction.
3. Benchmarking Utility: The nuScenes-format conversion and baseline results provide a standardized testbed for future research in multi-view in-cabin perception.
4. Modest Limitations: The authors acknowledge that while the pipeline is effective, it is not failure-free; identity merging/splitting can occur under heavy occlusion, and label noise remains at tight detection thresholds. They note that the dataset is currently limited to a single vehicle type and stationary scenarios, suggesting that future work should focus on increasing diversity and generalization.