Large-Scale Universal Defect Generation: Foundation Models and Datasets

1. Problem Statement

Existing methods for generating industrial defects/anomalies face three critical limitations:

• Data Scarcity & Overfitting: Most approaches rely on few-shot learning due to the lack of large-scale paired defect editing data. This causes models to overfit to specific defect categories, resulting in poor generalization to unseen objects or scenarios.
• Morphological Variability: Defects vary significantly in scale, shape, and texture. Learning from a few samples fails to capture the underlying distribution, leading to unrealistic synthesis and poor consistency with reference images.
• Limited Generalization in Pre-trained Models: General-purpose image editing models (e.g., advanced text-to-image or inpainting models) lack specific prior knowledge about industrial defects, making it difficult to synthesize rare or complex defects even with detailed text prompts.
• Pipeline Complexity: State-of-the-art zero-shot methods often require external Multi-Modal Large Language Models (MLLMs) and large offline repositories, making them costly and difficult to scale.

2. Methodology

The authors propose UniDG, a universal foundation model for defect generation, supported by a new large-scale dataset UDG. The methodology consists of four core components:

A. The UDG Dataset (300K Samples)

To address the lack of training data, the authors constructed UDG, a dataset of 300K normal-abnormal-mask-caption quadruplets spanning industrial, natural, and medical domains.

• Construction Pipeline: A multi-agent system was used:
  1. Inpainting Agent: Uses a Flux.1-based model to recover "normal" counterparts from abnormal images and masks.
  2. Captioner Agent: Uses advanced MLLMs to generate structured, fine-grained captions (global, object-level, and defect-level) conditioned on the normal-abnormal-mask triplets.
  3. Verifier Agent: Filters low-quality quadruplets by checking normality, caption consistency, and defect existence.
• Scale: Aggregates 50 public datasets, covering 269 original defect types mapped to 28 standardized categories.

B. UniDG Model Architecture

UniDG is built on the MM-DiT (Multimodal Diffusion Transformer) architecture, adapted for defect generation via a Defect-Context Editing strategy.

• Adaptive Defect Crop: Isolates the defect from the reference image with an appropriate amount of context, scaling the crop based on defect size to preserve high-frequency details.
• Structured Diptych Input: The model receives a concatenated input of the cropped reference defect and the masked source image ($I_{diptych} = [I_{cropped\_ref}; I_{masked\_src}]$).
• Multimodal Fusion:
  • Image Branch: Encodes the diptych and mask via a VAE.
  • Condition Branch: Combines text features (for instruction-based editing) and semantic features from the reference defect (extracted via SigLIP).
  • Attention Mechanism: Uses MM-DiT to fuse these conditions, allowing the model to transfer defect semantics from the reference to the target region while preserving the background.

C. Two-Stage Training Strategy

To balance diversity and consistency, a two-stage training recipe is employed:

1. Diversity-SFT (Supervised Fine-Tuning): Trains on three complementary distributions to maximize diversity:
   • Single Object Defect-Fill: Same defect, same object (ensures identity adherence).
   • Single Object Intra-Defect Cross-Sampling: Same object, different defect instances (ensures category consistency with varied details).
   • Multi-Object Intra-Defect Cross-Sampling: Different objects, same defect type (enables cross-object transfer).
2. Consistency-RFT (Reinforcement Fine-Tuning): Uses Flow-GRPO (Group Relative Policy Optimization) to align the model with human preferences.
   • Reward Models: Two specialized reward models guide the RL process:
     • Defect-Und-Reward: An MLLM-based model evaluating reference adherence, consistency, reasonability, and quality.
     • Defect-Recog-Reward: A task-specific model evaluating defect classification and segmentation accuracy.

3. Key Contributions

1. UDG Dataset: The first large-scale (300K), high-quality dataset of normal-abnormal-mask-caption quadruplets, enabling multimodal anomaly detection and generation research.
2. UniDG Foundation Model: A universal model capable of reference-based defect generation and text-instruction-based defect editing without per-category fine-tuning.
3. Novel Training Strategy: The combination of Diversity-SFT and Consistency-RFT with specialized reward models significantly improves both synthesis diversity and reference consistency.
4. State-of-the-Art Performance: Demonstrates superior performance in both image synthesis quality and downstream anomaly detection/localization tasks compared to few-shot and zero-shot baselines.

4. Experimental Results

Extensive experiments were conducted on MVTec-AD and VisA datasets.

• Downstream Anomaly Detection:
  • UniDG-RFT achieved 98.4% AUROC-I and 99.0% AP-I on MVTec-AD for single-class detection, outperforming all few-shot baselines (e.g., SeaS, AnoGen) and image insertion methods (e.g., AnyDoor).
  • In multi-class semantic segmentation, UniDG-RFT improved FG-mIoU by 20.5% over the previous best (AnoGen), demonstrating superior ability to distinguish and localize multiple defect categories simultaneously.
• Synthesis Quality:
  • Evaluated via MLLM-based metrics (Reference Adherence, Consistency, Reasonability, Quality), UniDG-RFT scored highest (e.g., 4.24/5.0 for Quality), significantly outperforming proprietary models like Gemini-3-Pro and open-source baselines.
  • Qualitative results show UniDG preserves texture, lighting, and morphology better than competitors, avoiding artifacts common in few-shot methods.
• Ablation Studies:
  • Confirmed that the Diptych input, Normal Regularization, and Defect Attention mechanisms are critical for performance.
  • Showed that Consistency-RFT is essential for stabilizing reference adherence, while Diversity-SFT is crucial for generalization.

5. Significance

• Scalability: UniDG eliminates the need for per-category fine-tuning and external MLLM repositories during inference, making it a practical, scalable solution for industrial inspection.
• Data Efficiency: By generating high-quality synthetic defects, it alleviates the data bottleneck for training supervised anomaly detectors, particularly for rare defect types.
• Foundation for Future Research: The UDG dataset and UniDG model establish a new baseline for universal defect generation, bridging the gap between natural image editing and industrial anomaly synthesis. The code and dataset are open-sourced to facilitate further research in multimodal anomaly detection.