Contrastive-SDXL: Annotation-Preserving Night-Time Augmentation for Pedestrian Detection

Problem Statement

Night-time pedestrian detection is a critical challenge for autonomous vehicles and Advanced Driver-Assistance Systems (ADAS). While detectors perform well in daytime conditions, their reliability degrades significantly in low-light environments due to reduced visibility, glare, noise, and large illumination differences. Collecting and annotating large-scale night-time datasets is costly and difficult to scale.

Cross-domain augmentation offers an alternative by translating labelled daytime images into night-style images, allowing the reuse of original annotations. However, existing diffusion-based image-to-image translation methods often prioritize perceptual realism over the preservation of specific object-level details. In safety-critical tasks like pedestrian detection, if a translation distorts, removes, or shifts a pedestrian, the inherited bounding-box annotations become unreliable, potentially causing the downstream detector to learn incorrect cues. There is a gap in using Latent Diffusion Models (LDMs) for augmentation that explicitly guarantees the preservation of small, occluded, and cluttered pedestrians required for valid detection training.

Methodology: Contrastive-SDXL

The authors propose Contrastive-SDXL, a day-to-night augmentation framework built on SDXL-Turbo and fine-tuned using Low-Rank Adaptation (LoRA). The framework is designed to translate daytime images into realistic night-time images while strictly preserving semantic structure and pedestrian integrity.

The method is guided by four complementary objectives and a two-stage data curation pipeline:

1. Generator Backbone and Efficient Adaptation:

   • The framework utilizes SDXL-Turbo, a one-step pretrained latent diffusion model, as the generator backbone.
   • To adapt the model to the night-time domain without full retraining, LoRA is applied to the UNet cross-attention layers, transformer MLPs, convolutional layers, and VAE encoder/decoder layers.
   • Encoder-decoder skip connections are retained to preserve fine-grained spatial structure.

2. Patch-Wise Semantic Contrastive Supervision (DINOv2-guided):

   • Unlike previous unpaired translation methods (e.g., PSCUT) that rely on generator encoder features (which can be unstable in diffusion models due to stochastic noise), Contrastive-SDXL uses a frozen, pretrained DINOv2 encoder.
   • This external encoder extracts multi-level semantic features from both the source (day) and translated (night) images.
   • Two losses are computed on these features:
     • Semantic Relations Consistency Loss (SRC): Preserves the relative semantic relationships between patches in the source image and their counterparts in the translated image using Jensen-Shannon divergence.
     • Hard Negative Contrastive Loss (hDCE): Pulls corresponding source and translated patches closer while pushing non-matching, semantically similar patches apart to enforce distinctiveness.

3. Detector-Guided Object Consistency Loss:

   • To explicitly ensure pedestrians remain detectable, a pretrained YOLO11 detector (fine-tuned on EuroCity Persons) is used as a fixed supervisor.
   • The generator minimizes a detection loss ($L_{det}$) computed on the translated image against the original daytime bounding box annotations. This penalizes translations where pedestrians are distorted, erased, or misaligned, ensuring the original annotations remain valid.

4. Adversarial Alignment and Identity Regularization:

   • An adversarial loss ($L_{adv}$) with a discriminator augmented by CLIP features ensures the generated images align with the statistical distribution of real night-time scenes.
   • An identity loss ($L_{idt}$) encourages the model to make minimal changes to images that are already in the target domain.

5. Two-Stage Data Curation Pipeline:

   • To ensure synthetic data reliability, a filtering process is applied before training detectors:
     • Stage 1 (Fidelity): Filters out translations where the DINOv2 feature similarity between source and target falls below a threshold (0.95), ensuring semantic faithfulness.
     • Stage 2 (Pedestrian Preservation): Uses a YOLOv8-based binary classifier to verify that pedestrian patches cropped from the translated image (at original annotation locations) are still classified as pedestrians. Images failing this check are discarded.

Key Contributions

• Investigation of LDMs: The work assesses pretrained Latent Diffusion Models as cross-domain augmentation tools for safety-critical night-time pedestrian detection.
• Contrastive-SDXL Framework: Proposes a novel day-to-night translation framework using SDXL-Turbo and LoRA, guided by DINOv2-based patch-wise semantic contrastive learning rather than generator features.
• Object Consistency Loss: Introduces a detector-guided loss to explicitly enforce pedestrian preservation, ensuring annotation validity during translation.
• Empirical Validation: Demonstrates that consistency-driven diffusion augmentation can effectively support safety-critical detection, achieving performance comparable to training on real night-time data.

Experimental Results

The method was evaluated using the EuroCity Persons (ECP) dataset and tested on two downstream detectors: Pedestron and YOLO26.

• Image Quality: Contrastive-SDXL achieved a Fréchet Inception Distance (FID) of 22.57 and a Wasserstein Distance (WD) of 11.27 against real ECP night-time images. This significantly outperformed baselines like InstructPix2Pix (FID 64.11) and CycleGAN-Turbo (FID 38.36). UMAP visualizations confirmed that synthetic images clustered closer to real night-time data than other methods.
• Detection Performance:
  • Detectors trained with Contrastive-SDXL synthetic images (without any real night-time data) achieved a 6–7% reduction in miss rate compared to a daytime-only baseline.
  • Specifically, for the Pedestron detector, the "All" miss rate on the night-time validation set dropped from 20.0% (daytime baseline) to 16.0% using only synthetic data.
  • When mixed with just 20% real night-time data, Contrastive-SDXL achieved a miss rate of 14.0%, matching the performance of a model trained on 100% real night-time data.
  • Similar improvements were observed with the YOLO26 detector, demonstrating generalization across architectures.
• Generalization: The method showed the ability to generalize to out-of-distribution urban scenes (TJU-DHD dataset), preserving pedestrian structure in unfamiliar environments.

Significance and Claims

The paper claims that Contrastive-SDXL demonstrates that consistency-driven diffusion augmentation is a viable and effective strategy for safety-critical night-time pedestrian detection. The authors emphasize that for safety-critical perception, visual realism alone is insufficient; the preservation of object-level semantics and annotation validity is paramount.

The results suggest that:

1. Pretrained LDMs can serve as powerful augmentation tools if guided by explicit consistency constraints (semantic contrastive and object-level losses).
2. Synthetic data generated by this framework can significantly reduce the domain gap between day and night, approaching the performance of models trained on real night-time data with minimal real data injection.
3. The proposed framework is not tied to a specific detector architecture, as benefits were observed across both Pedestron and YOLO.

The authors conclude that while the method shows promise, future work should explore its application to other transportation tasks (e.g., traffic sign recognition) and investigate how detector architecture and fine-tuning strategies affect adaptation performance.