SADGE: Structure and Appearance Domain Gap Estimation of Synthetic and Real Data

Technical Summary: SADGE – Structure and Appearance Domain Gap Estimation

Problem Statement
The generation of synthetic data is a critical strategy for scaling computer vision systems, particularly when real-world annotations are expensive or unavailable for edge cases. However, a fundamental bottleneck remains: predicting whether a specific synthetic dataset will successfully transfer to real-world performance before incurring the cost of downstream model training. Current evaluation practices rely heavily on trial-and-error, where practitioners generate datasets, train models, and measure real-world performance to validate rendering pipelines or domain-randomization strategies.

Existing metrics for evaluating synthetic data quality primarily focus on Appearance Similarity (e.g., PSNR, FID, CLIP, LPIPS) to measure semantic or pixel-level alignment between real and synthetic images. While some approaches consider Geometric Similarity (structural alignment), there is no established evidence regarding which similarity metric best predicts downstream task performance across diverse synthetic datasets. Furthermore, it is unclear whether appearance or geometry alone is sufficient, or if their interaction is the governing factor for synthetic data utility.

Methodology: SADGE
The authors propose SADGE (Structure and Appearance Domain Gap Estimator), a zero-shot, training-free metric designed to estimate the utility of synthetic image datasets for downstream visual recognition tasks. SADGE operates by jointly modeling two complementary signals:

1. Appearance Similarity ($A$): Computed using pretrained visual encoders (e.g., DINOv3, CLIP, SigLIP, SAM) to measure high-level semantic and textural alignment. The score is defined as the cosine similarity between normalized image embeddings.
2. Geometric Consistency ($G$): Computed using geometry-aware matchers (e.g., MASt3R, LoFTR, SuperPoint+LightGlue) to measure structural fidelity. This involves estimating dense or semi-dense correspondences between real and synthetic images and counting the number of geometrically valid inliers (verified via epipolar geometry or 3D consistency).

Fusion Mechanism
SADGE does not simply average these scores. Instead, it fuses them using a constrained bilinear interaction model:
$$\text{SADGE} = a\hat{G} + b\hat{A} + c\hat{G}\hat{A}$$
Where $\hat{A}$ and $\hat{G}$ are z-score normalized appearance and geometry scores, respectively. The coefficients $a, b, c \geq 0$ are learned to maximize the correlation with downstream performance. This formulation enforces monotonicity (improving either score should not decrease the total) and complementarity (the interaction term $c\hat{G}\hat{A}$ ensures the score is highest when both appearance and geometry are strong). The model assumes that synthetic data is most useful when it is both visually realistic and structurally faithful.

Experimental Protocol
The evaluation spans five public synthetic-to-real benchmark families and 15 dataset-level variants, totaling approximately 79,000 image pairs. The benchmarks cover diverse domains and tasks:

• DIMO: Industrial metal objects (6D pose estimation).
• Virtual KITTI 2: Driving scenes (semantic segmentation).
• TUD-L / BOP: Textureless objects (6D pose estimation).
• RarePlanes: Aerial imagery (object detection).
• ASD (Agricultural Synthetic Dataset): Crop/weed segmentation.

The authors compare SADGE against a wide range of baselines, including pure appearance metrics (FID, LPIPS, PSNR, CLIP, DINO variants) and pure geometry metrics (SSIM, LoFTR, SuperPoint+LightGlue).

Key Results

• Superior Correlation: SADGE achieves the strongest association with downstream transfer performance among all tested metrics. It reaches a Pearson correlation of $r = 0.879$ and a Spearman rank correlation of $\rho = 0.768$ across the 15 dataset variants.
• Comparison to Baselines: SADGE significantly outperforms the strongest geometry-only baseline (MASt3R, $r = 0.677$) and the strongest appearance-only baseline (LPIPS, $r = 0.649$).
• Robustness: In leave-one-dataset-out evaluations, SADGE consistently ranks first. The performance drop is most significant when the DIMO dataset (which emphasizes structural variation) is excluded, suggesting that geometry is a critical predictor in structurally demanding scenarios, while appearance dominates in scenarios with high illumination or texture variation.
• Optimal Configuration: The best-performing configuration fuses DINOv3 (appearance) with MASt3R (geometry).
• Sensitivity: The fusion coefficients are robust; the high correlation is maintained across a broad plateau of coefficient values, indicating the result is not an artifact of overfitting to a narrow parameter space.

Significance and Claims
The paper claims that synthetic data utility is not governed by a single factor (appearance or geometry) but by their non-linear interplay. SADGE provides a principled, training-free method to rank candidate synthetic datasets, allowing practitioners to:

• Compare rendering configurations, domain-randomization schedules, and generative pipelines before expensive training.
• Identify which synthetic variants are likely to transfer well to real-world tasks.
• Avoid the "trial-and-error" loop of training models solely to evaluate dataset quality.

The authors emphasize that SADGE is a proxy ranking metric, not a substitute for final real-domain validation, particularly in safety-critical applications. The study is limited by the availability of public benchmarks with comparable downstream evaluation protocols, but the sensitivity analysis suggests the metric's interaction model generalizes beyond the specific datasets tested.