HarmoniDiff-RS: Training-Free Diffusion Harmonization for Satellite Image Composition

1. Problem Definition

The paper addresses the challenge of Satellite Image Composition, defined as the task of seamlessly integrating a rigid source region (e.g., a building, road, or port) into a target satellite scene. Unlike natural image composition, which often allows for non-rigid geometric deformation (changing poses or shapes) to fit the context, satellite composition requires geometric rigidity. The primary challenges are:

• Radiometric Consistency: Ensuring the source matches the target's illumination, color, and spectral characteristics (e.g., seasonal or sensor differences).
• Boundary Harmonization: Eliminating hard seams between the source and target without distorting the original structure.
• Lack of Supervision: Satellite imagery rarely provides precise segmentation masks, making traditional learning-based methods that rely on accurate delineation difficult to apply.
• Inapplicability of Existing Methods: Standard natural image harmonization methods often deform geometry or require precise masks, while inpainting models struggle to adapt the radiometric properties of the source to the target domain.

2. Methodology: HarmoniDiff-RS

The authors propose HarmoniDiff-RS, a training-free framework that operates entirely within the latent space of a pre-trained Diffusion model (specifically DiffusionSat). The method consists of three core components:

A. Latent Mean Shift (LMS)

To align the radiometric characteristics of the source with the target without training, the method performs a channel-wise mean shift in the latent space.

• Mechanism: It calculates the mean statistics of the target latent and the source latent. It then shifts the source latent so its channel-wise means match the target.
• Goal: This transfers global style, illumination, and color tones from the target scene to the source patch, ensuring radiometric consistency.

B. Timestep-wise Latent Fusion

The authors observe a trade-off in Denoising Diffusion Implicit Model (DDIM) inversion:

• Early Timesteps: Produce highly harmonious results (good blending) but lose structural identity (content distortion).
• Late Timesteps: Preserve the original structure and semantics of the source but result in visible, hard boundaries.
• Strategy: The method generates composite candidates at various timesteps. It then fuses these candidates using an edge-aware mask:
  • Boundary Regions: Uses latents from early timesteps to ensure smooth, harmonious transitions.
  • Interior Regions: Uses latents from late timesteps to preserve the geometric integrity and semantic content of the source.

C. Harmony Classifier & Selection

Since different timesteps yield different trade-offs between harmony and fidelity, the system needs an automatic selection mechanism.

• Classifier: A lightweight ResNet-18 is trained to predict a "Harmony Score" for a given composite image and mask.
• Selection: The framework generates multiple candidates (via the fusion strategy) and selects the one with the highest predicted harmony score as the final output.

3. Key Contributions

1. Novel Task Definition: The paper formally defines satellite image composition as a rigid, structure-preserving task distinct from natural image composition, emphasizing boundary harmonization over semantic deformation.
2. Training-Free Framework: Introduces HarmoniDiff-RS, which leverages generative priors in the latent space to achieve harmonization without fine-tuning or online optimization.
3. Hybrid Fusion Mechanism: Proposes a Timestep-wise Latent Fusion strategy that effectively balances the complementary strengths of early (harmonious) and late (structurally faithful) inverted latents.
4. Benchmark Dataset (RSIC-H): Constructs a new benchmark dataset derived from fMoW containing 500 paired composition samples with diverse cross-domain variations (season, sensor, illumination).
5. New Metrics: Introduces Harmony Score (HS) and Boundary Gradient Difference (BGD) to quantitatively evaluate compositional realism and boundary smoothness.

4. Experimental Results

The method was evaluated against baselines including Copy-Paste, Poisson Blending, SD2 Inpainting, and FreeCompose.

• Qualitative Results: HarmoniDiff-RS produces seamless composites that adapt the source to the target's lighting and context while preserving the source's geometry. It successfully handles difficult cases where source and target have significant semantic or radiometric mismatches (e.g., adapting a port scene to a different season).
• Quantitative Results:
  • Harmony Score (HS): Achieved the highest score (0.225), indicating superior visual consistency.
  • Boundary Gradient Difference (BGD): Achieved the lowest score (4.88), indicating the smoothest transitions.
  • FID: Ranked second (92.38), slightly behind Poisson Blending. The authors attribute this to the inherent high-frequency loss in the VAE bottleneck of diffusion models, noting that this is a limitation of the backbone, not the harmonization strategy.
• Ablation Study: Confirmed that Latent Mean Shift improves fidelity, while the Timestep-wise Fusion is critical for balancing harmony and identity preservation.

5. Significance and Limitations

Significance:

• Scalability: The training-free nature allows the method to be applied immediately to various remote sensing tasks (data augmentation, disaster simulation, urban planning) without the computational cost of retraining models.
• Domain Adaptation: It effectively bridges the gap between different satellite sensors, acquisition times, and geographic regions.
• Future-Proof: The pipeline is designed to be plug-and-play with future latent diffusion models that offer better reconstruction capabilities.

Limitations:

• VAE Bottleneck: The method inherits the high-frequency smoothing artifacts of the underlying VAE, leading to a slight loss in fine texture details (e.g., roof patterns).
• Contextual Mismatch: Severe semantic mismatches between source and target can still result in unnatural structures.
• Shadow Control: The framework currently lacks explicit control over cast shadows, which can lead to physically inconsistent shading in some scenarios.

In conclusion, HarmoniDiff-RS represents a significant step forward in remote sensing synthesis by providing a robust, efficient, and training-free solution for integrating rigid objects into satellite imagery while maintaining geometric integrity and radiometric realism.