Technical Summary: Texture-Shape Bias Balancing for Robust Synthetic-to-Real Semantic Segmentation in Automotive NIR Imagery

Problem Statement

Semantic segmentation is critical for modern automotive visual perception, enabling pixel-level understanding for both exterior obstacle detection and interior occupant monitoring. While Near-Infrared (NIR) imaging offers stable detection across day-night cycles, developing robust domain-specific models is hindered by the scarcity of high-quality, annotated real-world NIR data. Synthetic datasets provide a scalable alternative but suffer from a significant "domain gap" when transferred to real-world scenarios.

A core challenge identified in this work is the inductive bias learned by deep networks. Models trained on synthetic or stylistically homogeneous data often exhibit a texture bias, relying on local statistical regularities rather than global shape cues. This reliance causes poor generalization when the texture statistics change between the synthetic source and the real NIR target domain. Furthermore, existing domain adaptation (DA) and domain generalization (DG) methods are primarily designed for RGB imagery and do not account for the specific spectral and textural characteristics of NIR data, where synthetic simulators are often designed for the visible spectrum.

Methodology

The authors propose a generative augmentation framework designed to bridge the synthetic-to-real (synth2real) gap in automotive NIR segmentation by explicitly balancing texture and shape biases during training. The framework consists of two primary modules:

1. Target Style Adaptation (TSA)

To address the appearance gap between synthetic RGB images and real NIR imagery, the authors employ a Latent Diffusion Model (LDM) fine-tuned via Low-Rank Adaptation (LoRA).

• Process: A LoRA is trained on a small, curated set of unlabeled real NIR images to learn the target domain's style.
• Conditioning: To preserve semantic structure and prevent the degradation of safety-critical thin objects (e.g., seat belts, glasses), the generation is constrained by multi-signal conditioning using ControlNet. Signals include:
  • The original segmentation mask (to enforce semantic consistency).
  • Canny edges extracted from the input image (to preserve scene contours).
  • Additional structural cues (e.g., depth, surface normals) for exterior scenes.
• Outcome: Synthetic images are transformed into realistic NIR-style variants while maintaining the original ground truth annotations.

2. Voronoi-based Style Diversification (VSD)

To mitigate overfitting to specific texture statistics and encourage reliance on shape, the authors introduce a style diversification strategy.

• Mechanism: Images are partitioned into locally coherent but irregular regions using Voronoi tessellation.
• Stylization: Each Voronoi cell is independently stylized using Adaptive Instance Normalization (AdaIN), transferring texture from a style image (sampled from the Painter by Numbers dataset) to the content image.
• Grayscale Conversion: Since the target is NIR, color cues are suppressed via grayscale conversion to better approximate the NIR spectrum.
• Goal: This introduces stochastic, spatially varying texture perturbations while preserving the global geometric structure, forcing the model to rely less on dataset-specific textures.

The framework allows for modular application: TSA alone, VSD alone, or a combination of both.

Key Contributions

1. First Systematic Study: The paper presents the first comprehensive investigation of synth2real domain adaptation specifically for automotive NIR semantic segmentation, covering both interior (in-cabin) and exterior (road scene) scenarios.
2. Generative Augmentation Pipeline: A novel pipeline combining LoRA-based target style adaptation and Voronoi-based style diversification that enables effective training without annotated real NIR data.
3. Bias Balancing Analysis: A systematic investigation into how controlled style variations influence the texture-shape bias trade-off, demonstrating that balancing these biases improves cross-domain generalization.
4. Comprehensive Validation: Extensive evaluation across three distinct architectural paradigms (CNN-based DeepLabV3+, Transformer-based SegFormer, and hybrid Mask2Former) on proprietary interior data and public benchmarks (RANUS for real, GTA5 for synthetic).

Experimental Results

Domain Gap Reduction

Experiments on interior and exterior datasets demonstrate that the proposed augmentation strategies significantly reduce the domain gap:

• Interior Data: The best configurations closed 28.4% of the domain gap for DeepLabV3+ and 25.0% for SegFormer.
• Exterior Data: The method achieved a 63.6% reduction in the domain gap for DeepLabV3+ and 45.2% for SegFormer.
• Architecture Sensitivity:
  • DeepLabV3+ (CNN): Benefited most from mild, spatially localized stylization (e.g., VSD with 16 cells and 25% probability) combined with TSA.
  • SegFormer (Transformer): Showed similar gains with structured stylization.
  • Mask2Former (High-capacity Transformer): Performed best with fully stylized datasets (VSD(1,1.0)), suggesting its architecture is more robust to global appearance changes.

Bias and Robustness Analysis

• Cue Decomposition: Using Edge-Enhancing Diffusion (EED) to remove texture and Voronoi Patch Shuffling (VPS) to remove shape, the authors quantified model reliance. Synthetic-only training resulted in texture-dominated models (low shape bias scores). The proposed augmentations systematically shifted models toward shape-based representations, aligning them closer to real-world baselines.
• Corruption Robustness: Models trained with the proposed augmentations showed significantly higher robustness to distortions (low-pass filtering, uniform noise, elastic deformation, swirl) compared to the synthetic baseline. The optimal augmentation for robustness varied by architecture (e.g., mild stylization for CNNs vs. full stylization for Transformers).
• Class-Specific Performance: Gains were most pronounced for classes with stable geometry (e.g., seat components, road, sky), with improvements up to 52 IoU points for DeepLabV3+. Fine-grained or small classes (e.g., glasses, hair) showed mixed results, highlighting remaining challenges with high-frequency details.

Significance and Claims

The paper claims that inductive bias balancing through generative data augmentation is an effective strategy for bridging the synth2real gap in NIR semantic segmentation. By explicitly compensating for texture bias during training, the method produces models that generalize better to real-world scenarios without requiring expensive pixel-level annotations of real NIR data.

The authors emphasize that this approach is model-agnostic and relies only on a small set of unlabeled target images for style adaptation, making it easily integrable into existing automotive training stacks. The results suggest that for safety-critical applications like driver monitoring and autonomous driving, rebalancing texture and shape cues is essential for robustness, particularly when transferring from synthetic RGB simulations to real NIR sensors.

Note: The authors explicitly state that while gains are significant for geometrically stable classes, fine-grained categories remain challenging, and the method does not claim to solve all domain shift issues, particularly for small, high-frequency objects.