A Vendor-Agnostic LiDAR Data Conversion System with Multi-Signal Detection and Multi-Format Output

Technical Summary: A Vendor-Agnostic LiDAR Data Conversion System with Multi-Signal Detection and Multi-Format Output

Problem Statement
The paper addresses the fragmentation inherent in processing raw LiDAR data from different manufacturers (specifically Ouster, Velodyne, Hesai, and Livox). While these sensors output data in PCAP (Packet Capture) files, their internal structures—including UDP port conventions, packet layouts, magic bytes, and encoding schemes—are mutually exclusive. Consequently, existing tools lack cross-vendor capability; a pipeline built for one vendor cannot process data from another without significant re-engineering. Furthermore, the ecosystem suffers from "toolchain friction":

• SDK Dependencies: Vendors provide proprietary Software Development Kits (SDKs) with unique runtime environments and language assumptions. Notably, Hesai and Livox lack production-grade, open-source Python SDKs, forcing engineers to write custom parsers.
• Format Incompatibility: Downstream applications (robotics, machine learning, geomatics) require diverse standard formats (PCD, LAS, LAZ, BIN, CSV), necessitating multiple conversion steps.
• Manual Configuration: Current workflows require users to manually identify the sensor vendor before selecting the appropriate parser, creating a bottleneck in automated or batch processing scenarios.

Methodology
The proposed system is a unified pipeline designed to ingest raw PCAP files, automatically identify the sensor vendor, and convert the data into industry-standard formats without user intervention. The architecture is divided into three layers:

1. Orchestration Layer: Manages execution state and passes input parameters to downstream modules.

2. Detector Layer (Vendor Identification): This module employs a multi-signal scoring mechanism to identify the vendor directly from the raw bitstream. It evaluates six independent signals, assigning weighted scores to determine the source sensor:

   • File Extension (0.5): Checks for .pcap, .lvx, or .rxp.
   • Magic Bytes (3.0): Scans the first 1,024 bytes for manufacturer signatures (e.g., 0xFFEE for Velodyne, 0x0001 for Ouster).
   • Companion Files (2.5): Validates the presence of sidecar metadata (e.g., Ouster's .json calibration files).
   • UDP Port Detection (3.5): The highest-weighted signal, analyzing destination ports in the first 10 UDP packets (e.g., 2368/2369 for Velodyne, 7502/7503 for Ouster).
   • Packet Structure (3.0): Matches the first 2 bytes of the UDP payload against vendor-specific magic bytes.
   • Packet Size (2.0): Samples the first 50 payloads to verify if they fall within vendor-specific size ranges.
   • A vendor is selected if its cumulative score exceeds a defined threshold (2.0), ensuring high confidence.

3. Conversion Layer:

   • Base Wrapper: An abstract class enforcing a standardized interface (e.g., convert_to_las, get_vendor_info) to decouple the core logic from specific vendor implementations. It handles serialization to PCD, BIN, and CSV, and manages LAZ compression via a fallback chain (laspy or external laszip).
   • Vendor-Specific Wrappers:
     • Ouster & Velodyne: Utilize native C++ SDKs (ouster-sdk, velodyne-decoder) with Python bindings. These leverage pre-computed lookup tables and internal C++ processing for high throughput.
     • Hesai & Livox: Rely on purely Python-based parsing using the dpkt library, as no open-source C++ SDKs are available. For Livox, the system also includes a custom bit-level parser for proprietary LVX/LVX2 formats using Python's struct module.

Key Contributions

• Automatic Vendor Detection: The system eliminates the need for manual vendor flags by deriving the sensor identity from raw packet characteristics using a weighted multi-signal approach.
• Unified Multi-Format Output: A single command converts raw PCAP data into five standard formats: LAS, LAZ, PCD, BIN, and CSV.
• Hybrid Parsing Architecture: The system integrates high-performance C++ SDKs where available (Ouster, Velodyne) while providing functional Python-based parsers for vendors lacking open SDKs (Hesai, Livox), all within a single cohesive framework.
• Consumer-Grade Compatibility: The pipeline is designed to run on standard hardware (tested on an Intel Core i3 with 8GB RAM) without requiring specialized infrastructure or complex environment setups.

Results and Performance
The system was evaluated on real-world outdoor datasets from all four supported vendors.

• Throughput Disparity: A significant performance gap was observed between C++-backed and Python-based parsers.
  • Ouster: Peaked at 2.08 million points/second.
  • Velodyne: Peaked at 1.47 million points/second.
  • Hesai: Averaged approximately 110,000 points/second.
  • Livox: Averaged approximately 150,000 points/second.
• Efficiency Gap: The C++ parsers demonstrated an 8–10× higher throughput compared to the Python-based parsers. The paper attributes this to the overhead of Python-level packet decoding and memory handling, rather than the conversion logic itself.
• Scalability: While all vendors showed consistent conversion completeness, throughput degradation was observed at larger workloads, particularly for Ouster and Hesai, due to increasing memory pressure and I/O overhead during LAS file writing.
• Reliability: Vendor detection succeeded on all datasets without manual configuration, and no parser failures were recorded during the evaluation runs.

Significance
The paper positions this system as a solution to the "vendor heterogeneity" problem at the data ingestion stage. By automating vendor detection and standardizing the output format, the pipeline removes the engineering overhead of maintaining multiple disconnected processing workflows. It enables researchers and practitioners to process multi-vendor LiDAR data using a single, unified toolchain, facilitating the creation of diverse datasets for training and mapping without prior knowledge of the sensor manufacturer. The authors note that while the system bridges the gap in raw data decoding, it operates upstream of tools like PDAL, which handle subsequent processing on already decoded data.