🤖 AI

MAGIC: Few-Shot Mask-Guided Anomaly Inpainting with Prompt Perturbation, Spatially Adaptive Guidance, and Context Awareness

The paper proposes MAGIC, a few-shot anomaly generation framework that leverages Gaussian prompt perturbation, spatially adaptive guidance, and context-aware mask alignment to produce high-fidelity, diverse anomalies for robust industrial quality control.

Original authors: JaeHyuck Choi, MinJun Kim, Je Hyeong Hong

Published 2026-04-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: JaeHyuck Choi, MinJun Kim, Je Hyeong Hong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a quality control inspector at a factory. Your job is to spot defects on products like screws, pills, or circuit boards. The problem? You have thousands of pictures of perfect products, but almost no pictures of broken ones. You can't train your AI to spot a broken screw if it has never seen one.

To solve this, scientists usually try to "fake" broken images using computers. But existing methods have two big problems:

The "Global" approach: It tries to invent a broken product from scratch. It often gets the defect right but accidentally ruins the rest of the product (like turning a perfect screw into a melted blob).
The "Mask" approach: It takes a perfect product and paints a defect onto a specific spot. But it's too rigid. It only learns to make one specific type of scratch, and if you tell it to paint a scratch in the wrong place (like on the head of a screw when it should be on the side), it creates a weird, unrealistic image.

Enter MAGIC (Mask-Guided Anomaly Inpainting with Prompt Perturbation, Spatially Adaptive Guidance, and Context Awareness). Think of MAGIC as a master forger who can create realistic, diverse "broken" products without messing up the good parts.

Here is how MAGIC works, using three simple tricks:

1. The "Creative Stretch" (Gaussian Prompt Perturbation)

Imagine you are teaching an artist to draw a "scratch." If you only show them one photo of a scratch, they will memorize it exactly and draw the same scratch every time. That's boring and not helpful.

MAGIC uses a technique called Gaussian Prompt Perturbation. Think of this as giving the artist a slightly blurry, fuzzy instruction instead of a sharp, rigid one.

How it works: Instead of saying "Draw a scratch exactly like this," the system adds a little bit of "noise" or fuzziness to the instruction during both the learning phase and the drawing phase.
The Result: The artist learns the concept of a scratch rather than just copying one photo. This allows them to draw thousands of different scratches (long, short, deep, shallow) that all look real, rather than just repeating the same one.

2. The "Smart Brush" (Spatially Adaptive Guidance)

When you paint a defect onto a perfect product, you want the defect to look wild and varied, but you want the rest of the product to stay perfectly normal.

The Problem: Standard AI tools use the same "strictness" for the whole image. If they are too strict, the defect looks boring. If they are too loose, the background gets ruined.
The MAGIC Solution: They use a Spatially Adaptive Guidance brush.
- On the defect area: The brush is "loose." It encourages the AI to be creative and try many different textures and patterns for the scratch.
- On the background: The brush is "tight." It forces the AI to keep the background exactly as it was, ensuring the perfect parts of the screw or pill don't get distorted.
The Result: You get a highly varied, realistic defect that sits perfectly on a pristine background.

3. The "Context Detective" (Context-Aware Mask Alignment)

Sometimes, the user gives the AI a mask (a stencil) in the wrong place. For example, they might try to put a scratch on the very tip of a screw, but scratches usually happen on the head. If the AI just paints there, it looks fake.

MAGIC has a Context-Aware Mask Alignment module. Think of this as a detective who knows how objects work.

How it works: Before painting, the system looks at the object. It says, "Hey, you want to put a scratch here, but that's the smooth tip of the screw. Scratches usually happen on the head."
The Action: It automatically slides the stencil (the mask) to a semantically correct spot (the screw head) before painting.
The Result: The defect appears in a logical, realistic location, making the fake image indistinguishable from a real broken product.

The Big Picture

By combining these three tricks, MAGIC creates a massive library of "fake" broken products that are:

Diverse: They look different every time (not just copies).
Realistic: The defects look genuine, and the background stays perfect.
Logical: The defects appear in the right places.

The paper shows that when factories use these "fake" images to train their AI inspectors, the inspectors become much better at spotting real defects in the real world. MAGIC essentially teaches the AI what "broken" looks like by showing it a million different, realistic ways a product can break, without ever needing a real broken product to start with.

1. Problem Statement

In industrial quality control, Few-Shot Anomaly Generation (FSAG) is critical for training robust anomaly detection models when labeled defective data is scarce. Existing approaches generally fall into two categories, both of which have significant limitations:

Global Anomaly Generation (GAG): Methods that generate anomalies and masks simultaneously without input masks. While diverse, they often corrupt the background textures of normal regions, leading to unrealistic images.
Mask-Guided Anomaly Generation (MAG): Methods that use a normal image and a user-provided mask to inpaint anomalies. While they preserve the background, they suffer from two main issues:
1. Low In-Distribution Diversity: Models heavily conditioned on the normal image tend to memorize the few training examples, producing repetitive textures and shapes (mode collapse).
2. Semantic Misalignment: Generated anomalies often appear in semantically implausible locations (e.g., a scratch on a screw head when the mask was placed on the shaft), leading to unrealistic artifacts.

The core challenge is to generate high-fidelity, diverse anomalies that strictly adhere to the mask while ensuring the mask is placed in a semantically plausible region, all within a few-shot setting.

2. Methodology: MAGIC Framework

The authors propose MAGIC, a fine-tuned inpainting framework built upon the Stable Diffusion backbone. It introduces three complementary components to address the limitations of existing MAG methods:

A. Gaussian Prompt Perturbation (GPP)

Goal: To increase in-distribution diversity and prevent overfitting in few-shot settings.
Mechanism: Instead of applying noise only at inference (which causes distribution shifts), GPP injects Gaussian noise ( $\delta$ ) into the anomaly token embeddings (both for texture prompts and mask embeddings) during both training and inference.
Effect: This forces the model to learn a smooth, continuous manifold of realistic anomalies rather than memorizing discrete training points. At inference, sampling from this manifold yields diverse textures and mask shapes that remain within the target data distribution.

B. Spatially Adaptive Guidance (SAG)

Goal: To balance textural diversity in the anomaly region with background fidelity in the normal region.
Mechanism: Standard Classifier-Free Guidance (CFG) applies a uniform scale ( $w$ $w$ ). MAGIC applies spatially varying CFG scales:
- Normal Region: A high, fixed CFG scale ( $w_d = 7.5$ ) is used to strictly preserve the background consistency.
- Anomaly Region: A dynamic scale ( $w_a(t)$ ) is applied. It starts low ( $w_m = 4$ ) during early diffusion steps to encourage diverse texture exploration and gradually increases to $w_d$ via a cosine schedule as denoising progresses to refine realism.
Effect: This allows the model to explore diverse anomaly textures early on while ensuring the final output is realistic and the background remains untouched.

C. Context-Aware Mask Alignment (CAMA)

Goal: To resolve the issue of masks being placed in semantically implausible locations.
Mechanism:
1. Keypoint Extraction: From a reference anomaly pair (training data), three keypoints are selected: the mask centroid ( $p_c$ ) and two boundary points ( $p_u, p_\ell$ ) on a vertical line.
2. Semantic Matching: These keypoints are matched to the target normal image using a pre-trained semantic correspondence model (GeoAware-SC) to generate similarity maps.
3. Alignment: The optimal new centroid ( $q^*_c$ ) is determined by finding the point on the "semantic candidate line" (defined by the matched boundary points) that lies within the object's foreground mask and maximizes the similarity score.
4. Transformation: The input mask is translated to align with $q^*_c$ and clipped to the object's foreground.
Effect: Ensures anomalies are generated in structurally and semantically correct locations (e.g., a scratch on the head of a screw), preventing out-of-object artifacts.

3. Key Contributions

Novel Framework (MAGIC): The first few-shot MAG framework that simultaneously addresses low diversity, low fidelity, and semantic misalignment.
GPP Strategy: Demonstrates that perturbation must be applied during both training and inference to learn a smooth manifold, avoiding the distribution shift issues seen in inference-only noise injection.
Spatially Adaptive Guidance: A novel inference-time control mechanism that decouples guidance strength between anomaly and background regions, optimizing for both diversity and fidelity.
Semantic Mask Alignment: A robust module (CAMA) that leverages geometric and semantic cues to relocate masks, significantly improving the plausibility of generated defects.

4. Experimental Results

The authors evaluated MAGIC on four benchmark datasets: MVTec-AD, VisA, DAGM, and MVTec 3D-AD.

Generation Quality & Diversity:
- MAGIC achieved the lowest Kernel Inception Distance (KID) scores across all datasets, indicating superior alignment with the real anomaly distribution compared to baselines (AnoDiff, AnoGen, DualAnoDiff, SeaS).
- While IC-LPIPS (diversity metric) was slightly lower than some baselines, the authors argue this is due to MAGIC's strict adherence to in-distribution realism, avoiding the "artificial inflation" of diversity seen in methods that generate out-of-distribution artifacts.
Downstream Tasks:
- Anomaly Classification: MAGIC improved classification accuracy by 13.81% over the previous state-of-the-art (DualAnoDiff) on average.
- Detection & Localization: MAGIC set new state-of-the-art results in image-level (AUROC, AP, F1-max) and pixel-level (PRO) metrics across all four datasets.
Ablation Studies:
- Removing GPP led to significant drops in diversity and classification accuracy.
- Removing SAG reduced the balance between diversity and fidelity.
- Removing CAMA resulted in lower mask alignment scores and reduced classification performance, confirming the importance of semantic placement.

5. Significance

MAGIC represents a significant advancement in industrial anomaly detection by solving the "data scarcity" problem more effectively than previous generative approaches.

Practical Impact: By generating diverse, high-fidelity, and semantically correct training data, MAGIC enables downstream models to detect and classify defects more accurately, even with very few real-world examples.
Theoretical Insight: The paper highlights the critical importance of training-time perturbation for manifold learning in diffusion models and introduces a new paradigm for spatially adaptive control in inpainting tasks.
Robustness: The method proves robust across diverse object types (from 2D textures to 3D objects) and anomaly types, establishing a new benchmark for few-shot anomaly generation.