Multi-Conditioned Diffusion Synthesis of Sand Boils for Low-Resource Earthen-Levee Inspection

Technical Summary: Multi-Conditioned Diffusion Synthesis of Sand Boils for Low-Resource Earthen-Levee Inspection

Problem Statement
Earthen levees are critical infrastructure, yet their condition is difficult to monitor at scale. A primary safety-critical defect is the "sand boil," an early visible stage of internal erosion (piping) where pressurized water carries sand to the surface. Detecting and segmenting these defects from field imagery is hindered by a severe scarcity of annotated data. Defects appear only during high-water events, are distributed across vast linear networks, and require hydrologic expertise for accurate pixel-level annotation. Conventional data augmentation (rotations, flips, color jitter) fails to synthesize new scene distributions (e.g., different soil types, hydraulic regimes, or lighting) that are not already present in the small, curated reference sets.

Methodology
The paper proposes a diffusion-based synthesis pipeline designed to generate synthetic sand boil imagery while preserving defect geometry and ensuring label provenance. The pipeline consists of three coupled stages:

1. Backbone and Adaptation:

   • The system utilizes Stable Diffusion XL (SDXL) as the backbone. The authors deliberately chose SDXL over newer rectified-flow models (e.g., SD 3.5) due to the maturity of its publicly available ControlNet and IP-Adapter checkpoints and its lower tendency to memorize individual training frames at the specific adapter rank and data budget used.
   • DreamBooth with Low-Rank Adaptation (LoRA) is used to fine-tune the model on a small curated reference set (39 images). A rare trigger token (sbx) anchors the adapter, allowing the model to render sand boils without modifying the frozen base weights.

2. Multi-Conditioned Generation:

   • To constrain the generated scene to a real field reference while allowing textural variety, the pipeline employs a stack of four ControlNet branches: Canny edges (outline), monocular depth (3D layout), surface normals (dome-rim relief), and Holistically-Nested Edge Detection (HED) soft edges (rim texture).
   • Defect-Preserving Soft-Mask Inpainting: Instead of standard image-to-image or Poisson seamless cloning (which causes hard seams and color bleeding), the pipeline uses a soft-mask latent blending protocol. The ground-truth mask is inverted, eroded, and Gaussian-blurred to create a smooth gradient. During the diffusion loop, the denoiser predicts only within the gradient region, allowing the boundary to negotiate a coherent transition with the re-rendered background while preserving the defect core pixel-for-pixel.
   • Adaptive Geometry: The erosion radius and blur kernel width are derived adaptively from the defect's bounding box size, ensuring the preservation margin scales with the defect extent.

3. Prompt Curation and Quality Control:

   • Prompt Atlas: A taxonomy-driven system replaces hand-written prompt lists. A single JSON specification defines the domain concept, trigger token, and axes (e.g., soil type, weather, lighting). A builder expands this into a stratified prompt bank, filtered by length, de-duplicated via TF-IDF, screened for negative concepts, and validated against the reference set using CLIP image-text similarity.
   • Convex Hull Annotator (CHA): A fully automated, shape-aware pipeline converts probability maps into binary masks and polygon annotations. It routes components based on form factor (compact, elongated, or thin) to apply appropriate cleaning (convex hull, morphological closing, or skeletonization). This ensures a single, reproducible mask-production policy across all synthetic outputs.
   • Quality Filtering: A CLIP admissibility filter uses leave-one-out statistics from the reference set to define an acceptance band. It automatically rejects out-of-distribution (OOD) drift and memorized samples (near-duplicates of training frames) without human review.

Key Contributions
The paper introduces a generation pipeline with the following specific technical contributions:

• Defect-Preserving Soft-Mask Latent Blending: A per-step composition rule that pins the defect's appearance verbatim while allowing the denoiser to negotiate a coherent transition with the background, eliminating hard seams and color casts associated with binary inpainting and Poisson cloning.
• Defect-Size-Adaptive Soft-Mask Geometry: An automated rule that scales the soft-mask's erosion and blur parameters based on the defect's spatial extent, removing the need for per-image manual tuning.
• Convex Hull Annotator (CHA): A class-agnostic, shape-aware pipeline that produces uniform training labels and serves as a drift-checking quality gate for mask-conditioned generation.
• Mask-Conditioned Generation (MaskCN): A complementary path where a binary mask is the conditioning input, making the mask the ground-truth label by construction. This acts as a zero-annotation diversity engine, generating varied defect shapes and placements independent of real source geometries.
• Taxonomy-Driven Prompt Atlas: A reproducible, JSON-based prompt curation system that expands a single domain specification into a validated prompt bank, portable to new defect classes without code changes.
• Automated Reference-Set Similarity Filtering: A post-generation filter that removes OOD drift and memorization using auto-tuned thresholds derived from the reference set's leave-one-out CLIP statistics.

Results
The pipeline was evaluated on a dataset of 199 real training images and a curated reference set of 39 images.

• Dataset Generation: The pipeline produced 1,020 synthetic candidates; 815 passed the CLIP admissibility filter to form the augmented dataset.
• Image Quality: Evaluation used Fréchet Inception Distance (FID), Kernel Inception Distance (KID), CLIP image-text similarity, and LPIPS diversity.
  • No single preset dominated all metrics. The V4 preset (soft-mask inpainting with HED, surface normals, and IP-Adapter) was selected as the default for the augmented dataset. While it had a higher FID than some alternatives (indicating it ventured further from the tight real manifold), it offered superior label reliability and scene diversity.
  • MaskCN achieved the highest CLIP similarity and within-set diversity but failed the label-verification gate (median IoU drift of 0.15), meaning its "exact" labels could not yet be certified by the available segmenter checkpoint.
  • Poisson Baseline: A Poisson seamless-cloning baseline achieved lower FID scores but failed to provide genuine scene diversity or seam-free boundaries, confirming that low FID alone is insufficient for training data quality.
• Fidelity vs. Diversity: A precision/recall decomposition revealed that while some presets (like V1) had high fidelity (closeness to the real manifold), the selected V4 preset traded some fidelity for higher diversity and, crucially, a single-policy label provenance.
• Memorization Audit: A nearest-reference audit confirmed that no synthetic samples were CLIP-space duplicates of the training set (max similarity 0.925 vs. identity 1.0).

Significance and Claims
The paper modestly scopes its claims to image quality, label provenance, and diversity. It explicitly states that downstream segmentation performance is left to future work.

• The primary contribution is a methodology for generating high-quality, labeled synthetic data in a low-resource setting where annotated examples are scarce.
• The pipeline demonstrates that diffusion models can be adapted to niche scientific classes (earthen levee defects) without full retraining, provided that structural conditioning (ControlNet) and defect-preserving protocols (soft-mask inpainting) are employed.
• The authors emphasize the reproducibility of the system, releasing the complete pipeline, trained adapters, prompt atlases, and seed cascades.
• The work establishes a framework for "label-by-construction" (via MaskCN) and "label-by-reconstruction" (via soft-mask + CHA), offering a path to zero-annotation diversity engines, though the verification of these labels at scale remains a bottleneck.

The paper concludes that while no single configuration is optimal for all metrics, a curated mixture of complementary presets (balancing manifold coverage, diversity, and label reliability) constitutes the natural augmentation strategy for training downstream segmenters.