Automatic Labelling for Low-Light Pedestrian Detection

1. Problem Statement

Pedestrian detection is critical for the safety of Autonomous Vehicles (AVs) and Advanced Driver Assistance Systems (ADAS). While RGB cameras are the standard sensor, their performance degrades significantly in low-light or nighttime conditions due to low contrast, feature invisibility, and noise.

• The Bottleneck: Developing robust low-light detection models requires large quantities of labelled data. Manual annotation is time-consuming, expensive, and a major bottleneck.
• The Gap: Existing public datasets for low-light pedestrian detection are limited, and there is a lack of efficient, automated pipelines to generate labels for RGB images in these conditions.
• The Opportunity: Infrared (IR) cameras are largely invariant to visible light conditions. Pedestrians emit IR radiation (heat), making them easier to detect at night compared to RGB images. However, most AVs currently rely on RGB cameras.

2. Methodology

The authors propose a fully automated IR-to-RGB labelling pipeline that eliminates the need for manual human annotation. The process consists of three main stages:

A. Architecture Overview

1. Input: Paired unlabelled IR and RGB images (aligned and rectified).
2. IR Detection: A pre-trained IR pedestrian detection model (fine-tuned on daytime IR data) detects pedestrians in the IR frame. This model leverages the fact that IR detection generalizes well to low-light conditions.
3. Label Transfer: The bounding boxes detected in the IR frame are transferred directly to the corresponding RGB frame. This relies on the KAIST dataset's specific hardware setup where IR and RGB cameras are aligned via a beam splitter, ensuring pixel-level correspondence without complex 3D projection.
4. RGB Training: The transferred IR labels serve as "autolabels" to train standard RGB object detection models for low-light scenarios.

B. Experimental Setup

• Dataset: The KAIST Multi-spectral Pedestrian Detection Benchmark.
  • Used ~91.8k samples (excluding cyclists and grouped people to focus on single pedestrian detection).
  • Training: Daytime IR sequences used to train IR detectors; Low-light sequences used to train final RGB detectors.
  • Testing: Unseen low-light sequences (Sequence 09 for RGB, Sequence 02 for IR).
• Models Evaluated:
  • IR Detectors (for labelling): YOLOv11l, Faster RCNN, and RF-DETR.
  • RGB Detectors (final output): Same three architectures trained on the autolabels.
• Parameters: Two confidence thresholds (0.25 and 0.5) were tested to analyze the trade-off between label quantity and quality.

3. Key Contributions

1. Novel Pipeline: The first implementation of a fully automated pipeline that uses IR detections to generate labels for RGB low-light pedestrian detection, removing human effort entirely.
2. Scalability: Demonstrates a method to generate massive datasets for low-light training without manual annotation, addressing the data scarcity issue.
3. Performance Validation: Proves that models trained on these "noisy" autolabels can outperform models trained on high-quality human ground truth in specific metrics.
4. Architecture Analysis: Provides a comparative analysis of Transformer-based (DETR) vs. CNN-based (YOLO, RCNN) architectures in the context of low-light detection and autolabelling.

4. Results

The study evaluated performance using mAP@50, mAP@50-95, Precision/Recall, and Log Average Miss Rate (LAMR).

• Autolabel Quality:
  • IR-DETR generated the highest quality labels (highest F1 score and mAP).
  • Lowering the confidence threshold from 0.5 to 0.25 increased the number of generated labels significantly (e.g., IR-DETR generated ~49k labels at 0.25 vs. ~29k at 0.5) and generally improved the Recall of the final RGB models.
• RGB Detection Performance:
  • Superiority of Autolabels: Models trained on autolabels outperformed those trained on Ground Truth (GT) in 5 out of 6 cases for mAP@50 and LAMR, and in all 6 cases for mAP@50-95.
  • Best Configuration: The DETR architecture trained on autolabels generated by IR-DETR at 0.25 confidence achieved the best overall performance.
• Key Finding: The autolabelled models achieved higher mAP and lower LAMR than the GT-trained models.
  • Reasoning: The authors hypothesize that human GT labels include "hard cases" (heavily occluded or barely visible pedestrians) that act as label noise, confusing the model. The IR detector naturally filters these out (as it cannot detect them), effectively creating a "cleaner" training set of visible pedestrians.

5. Significance and Future Work

• Practical Impact: This method enables the creation of large-scale, low-light RGB detection datasets using only an IR-RGB sensor rig for data collection, while the final deployed system requires only a standard RGB camera.
• Limitations:
  • The approach currently relies on the KAIST dataset; validation on larger, more diverse datasets is needed.
  • The requirement for a calibrated IR-RGB sensor rig for data collection is expensive and complex, though only needed for the training phase.
  • Current models still struggle with extremely low recall (best recall ~0.37), indicating that low-light RGB detection remains a difficult challenge.
• Future Directions:
  • Expanding to other low-light tasks like pose estimation.
  • Testing state-of-the-art IR detection models to improve the initial label quality.
  • Gathering larger datasets to improve model robustness against occlusion and extreme darkness.

Conclusion: The paper successfully demonstrates that leveraging the lighting invariance of IR sensors to automate the labelling of RGB data is a viable and highly effective strategy. It not only reduces human labor but, counter-intuitively, produces detection models that outperform those trained on manual ground truth by filtering out ambiguous, low-visibility cases.