Technical Summary: Multi-Objective Optimization for Synthetic-to-Real Style Transfer

Problem Statement

Semantic segmentation networks require extensive pixel-level annotated data, which is prohibitively expensive to obtain for real-world images (e.g., ~90 minutes per Cityscapes image). While synthetic data from game engines like GTA5 offers automatic annotations, models trained on such data suffer from a "domain gap" when applied to real-world scenarios. Existing domain adaptation techniques, particularly those using diffusion models for style transfer, involve complex pipelines with numerous steps. Currently, the selection and ordering of these operations (e.g., darkening, sharpening, AdaIN, ControlNet) are determined manually. The combinatorial search space for effective data transformations is vast, making the identification of optimal style transfer pipelines difficult. Furthermore, evaluating candidate pipelines using standard distributional metrics (e.g., FID, CMMD) or by training segmentation models is computationally prohibitive for iterative optimization.

Methodology

The authors formulate the construction of style transfer pipelines as a Multi-Objective Optimization (MOO) problem. The goal is to find a sequence of transformations that simultaneously minimizes the domain gap (style similarity) while maximizing structural coherence (content preservation).

1. Problem Formulation (TSP Variant)

The pipeline selection is modeled as a modified Traveling Salesman Problem (TSP).

• Nodes: Each available data operator (e.g., Normalize, Brighten, AdaIN, CACTI, ControlNet) represents a "city."
• Terminal Node: A special STOP node is introduced to allow for variable-length solutions. The algorithm selects a subset of operators and an ordering, terminating when the STOP node is encountered. This formulation is analogous to the Orienteering Problem but without a strict budget constraint on the number of steps.
• Search Space: The solution space consists of permutations of operators, suitable for evolutionary search.

2. Multi-Objective Evolutionary Algorithm

The authors employ NSGA-II (Non-dominated Sorting Genetic Algorithm II) to evolve the pipelines.

• Population: Initialized via random permutation sampling.
• Operators: Edge recombination crossover and inversion mutation are used to generate offspring.
• Objectives:
  1. Minimize DISTS: Measures structural similarity between the source image and the transformed image to ensure ground-truth annotations remain valid.
  2. Minimize DreamSim: Measures perceptual style similarity between the transformed image and target domain samples (e.g., Cityscapes, ACDC).
• Evaluation Strategy: Instead of generating full datasets or training models for every candidate, the algorithm evaluates pipelines on a small set of paired images (5 source images). This drastically reduces computational cost while maintaining correlation with distributional quality.

3. Operators and Datasets

• Source: GTA5 dataset.
• Target: Cityscapes (clear day) and ACDC (adverse weather: night, snow, fog, rain).
• Transformation Set: Includes 6 traditional image processing operations (Blur, Brighten, Darken, Contrast, Sharpen, Normalize) and 3 deep learning-based style transfer operations (AdaIN, CACTI, ControlNet).

Key Contributions

1. Formulation: The paper proposes formulating style transfer pipeline selection as a combinatorial sequencing problem suitable for evolutionary optimization, treating it as a variable-length TSP variant.
2. Efficient Metrics: It identifies and validates the use of paired-image metrics (DISTS and DreamSim) as feasible surrogates for optimization. These metrics allow for rapid evaluation on small sample sizes, avoiding the high cost of distributional metrics or model retraining during the search process.
3. Diverse Solutions: The approach generates a Pareto front of non-dominated solutions, offering practitioners a range of trade-offs between content preservation and style adaptation rather than a single "best" pipeline.

Results

• Optimization Convergence: The evolutionary algorithm successfully converged to a Pareto front within 20 generations. The search space was found to be small enough to be thoroughly explored with a modest computational budget (13 hours total optimization time).
• Pipeline Characteristics:
  • Pipelines with higher style fidelity (lower DreamSim) tended to be more complex, involving multiple transformations and deep learning operators (e.g., ControlNet, CACTI).
  • Simple pipelines (e.g., single Sharpen operation) preserved content well but failed to adapt style.
  • The selected pipelines demonstrated a clear progression from synthetic appearance to realistic style across adverse weather conditions.
• Metric Correlation: DISTS and DreamSim values computed on the small evaluation set (5 images) remained highly consistent when applied to full generated datasets (1250 images), validating the efficiency of the paired-image evaluation strategy.
• Downstream Performance (Segmentation):
  • When trained on the generated datasets, the evolved pipelines achieved varying mIoU scores on ACDC and Cityscapes.
  • Surprising Finding: A baseline pipeline using only ControlNet outperformed most of the evolved multi-operator pipelines in terms of downstream segmentation mIoU.
  • The authors note that while the evolved pipelines achieved better perceptual style matching (lower DreamSim), this did not linearly correlate with improved segmentation performance. ControlNet's ability to maintain strong semantic coherence appeared more critical for the downstream task than the perceptual realism captured by DreamSim.

Significance and Limitations

The paper demonstrates that evolutionary algorithms can effectively automate the design of style transfer pipelines, providing diverse options for domain adaptation. The primary significance lies in the computational efficiency of the approach; optimizing a set of pipelines took roughly the same time as evaluating a single pipeline using full data generation, enabling rapid exploration of the design space.

However, the authors modestly acknowledge a limitation in their objective formulation. While the DISTS/DreamSim combination effectively assessed style transfer quality, it did not serve as a reliable surrogate for downstream segmentation performance. The evolved pipelines did not match the mIoU of a simple ControlNet baseline. The paper concludes that future work should focus on identifying metrics that can cheaply estimate downstream task performance, potentially leading to a three-objective formulation (Style, Segmentation Surrogate, and Compute Time) for more effective domain adaptation.