💻 computer science

Anomaly Image Generation under the Guidance of Knowledge Graph

This paper proposes a knowledge graph-guided framework that leverages prior knowledge about anomalies and their relationships to construct text prompts, enabling foundation models and diffusion techniques to generate realistic anomalous images for various product classes.

Original authors: Ylli Sadikaj, Lavdim Halilaj, Stefan Schmid, Hongkuan Zhou

Published 2026-06-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ylli Sadikaj, Lavdim Halilaj, Stefan Schmid, Hongkuan Zhou

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot artist how to draw "broken" versions of everyday objects, like a scratched car or a torn shirt. Usually, you have to sit down and manually write out hundreds of specific instructions (prompts) for the robot, saying things like, "Draw a red car with a scratch on the left door." This is slow, boring, and if you make a mistake, the robot draws something weird.

The paper you shared introduces a new tool called KG4IG to solve this problem. Think of it as giving the robot a smart recipe book (a Knowledge Graph) instead of just a blank notepad.

Here is how it works, broken down into simple steps:

1. The Smart Recipe Book (The Knowledge Graph)

Instead of guessing what a "scratch" looks like on a "car," the researchers built a digital encyclopedia (a Knowledge Graph) that maps out the rules of the world.

The Ingredients: It lists all the products (like cars, screws, or bottles).
The Problems: It lists all the ways those products can break (scratches, dents, missing parts).
The Rules: It knows the "laws of physics" for these items. For example, it knows that a "red scratch" on a "blue car" might look weird, or that a specific type of dent only happens on the metal part of a bottle, not the plastic cap.

This is like a master chef who knows exactly which spices go together and which ones ruin a dish. The robot doesn't have to guess; it just follows the recipe book.

2. The Sous-Chef (The Subgraph Extractor)

When you want the robot to draw a specific broken item, this "Sous-Chef" looks at your request and pulls out the exact page from the recipe book that applies.

If you ask for a "broken bottle," the Sous-Chef finds the section on bottles, looks at the rules for cracks, and checks the size and shape constraints.
It filters out impossible ideas (like a crack that is too big for the bottle) so the robot doesn't waste time trying to draw nonsense.

3. The Translator (The Prompt Generator)

Once the Sous-Chef has the right rules, the Translator turns those technical rules into natural language sentences that the robot artist understands.

Instead of just raw data, it writes a clear instruction: "Draw a glass bottle with a small, jagged crack near the base."
Because the recipe book has so many rules, the Translator can create thousands of unique, realistic instructions automatically, without a human needing to type them one by one.

4. The Artist (The Diffusion Model)

Finally, the robot artist (a "Diffusion Model") takes these new, high-quality instructions and paints the picture. Because the instructions were built using the "recipe book" rules, the resulting image of the broken object looks very realistic and follows the laws of how that object actually breaks.

Why is this a big deal?

In the world of computer vision, we often have thousands of pictures of perfect, normal items, but very few pictures of broken items. To teach computers to spot defects, we need more pictures of broken things.

The Old Way: Humans manually twist, flip, or crop pictures of broken items to make more. It's like trying to make a new cake by just rearranging the crumbs of an old one.
The New Way (KG4IG): The system uses the "recipe book" to invent brand new, realistic pictures of broken items from scratch. It understands the relationship between the object and the defect, ensuring the new images make sense.

In short: This paper proposes a system that uses a structured database of rules (a Knowledge Graph) to automatically write the instructions needed to teach AI how to draw realistic "broken" objects, saving humans from doing the tedious work of writing those instructions manually.

Technical Summary: Anomaly Image Generation under the Guidance of Knowledge Graph

Problem Statement

In the domain of anomaly detection, datasets are frequently characterized by a significant class imbalance, containing a vast majority of normal (defect-free) images and a scarcity of abnormal (defective) samples. Traditional approaches to address this imbalance rely on geometric image augmentations (e.g., rotation, flipping, scaling) or the manual crafting of text prompts (TPs) for generative models. However, manually constructing TPs that accurately describe anomalies, their locations, and sizes is a tedious, error-prone process that limits the diversity and quantity of generated samples. Furthermore, without prior expert knowledge, manually generated prompts may fail to respect the physical constraints and co-existence rules specific to certain product classes and defect types.

Methodology

The authors propose KG4IG, a framework designed to automatically construct a large volume of plausible text prompts by leveraging prior knowledge encoded in a Knowledge Graph (KG). The approach integrates a KG with a diffusion-based image generation pipeline. The methodology consists of the following core components:

1. Knowledge Graph for Anomalies (KGA)

The foundation of the approach is a domain-specific ontology constructed to represent products, their anomalies, and the relationships between them. The KGA is built upon three primary datasets: MVTec-AD, VisA, and MPDD. The ontology defines:

Entities and Classes: Including Product (comprising NormalProduct and DefectProduct), Defect, DefectType, and DefectCharacteristics (depth, shape, size).
Relationships: Formal definitions such as hasDefect, hasBoundingBox, and isAugmentedBy.
Constraints: Logical rules that enforce validity, such as preventing color augmentations that match the product color or restricting specific product types to only certain defect combinations (e.g., ProductTypeA can only have DefectTypeD1 or DefectTypeD2).
Inference: The system utilizes ontological definitions to infer new relation types not explicitly stated, enhancing the graph's expressiveness.

The ontology comprises 59 classes, 38 object properties, and 109 data properties, allowing for the validation of user prompts against domain constraints.

2. Prompt Generation Pipeline

The system automates the creation of text prompts through a four-step process:

Subgraph Extraction: From the KGA, the system extracts a subgraph ( $G_P$ ) containing the specific product class ( $P$ ), associated defect types ( $D_P$ ), and relevant attributes ( $D_A$ ) such as shape, size, depth, and location.
Constraint Filtering: The system applies constraints ( $C$ ) derived from the KG to eliminate invalid combinations, such as color conflicts or disallowed co-occurrences of defects.
Prompt Synthesis: For each valid configuration of defects and attributes, the system constructs a natural language description.
Diversification: Linguistic or structural variations are applied to generate a diverse set of prompts from the valid configurations.

3. Model Learning and Generation

The generated prompts serve as inputs for a text-to-image generative model (specifically a diffusion model). The model is trained to align textual and visual modalities, learning the hierarchical relationships between products and anomalies, their attributes, and contextual constraints. The ultimate goal is to synthesize anomalous images that respect the spatial semantic representation of the anomalies defined in the KG.

Key Contributions

Automated Prompt Construction: The paper introduces a method to replace manual prompt engineering with an automated system driven by a Knowledge Graph, capable of generating a "huge number" of plausible text prompts.
Domain-Specific Ontology: The development of a formal ontology that captures specific domain terminology, product classes, and the complex relationships and constraints governing anomalies.
Constraint-Aware Generation: The ability to enforce logical restrictions (e.g., co-occurrence rules, attribute conflicts) during the prompt generation phase, ensuring that the resulting text prompts are physically and logically consistent with the target domain.
Integration with Foundation Models: A pipeline that connects structured knowledge (KG) with modern diffusion models, enabling the generation of anomalies for objects of the same class or the augmentation of existing objects.

Results and Claims

The paper focuses primarily on the architectural design and the construction of the Knowledge Graph component rather than presenting extensive quantitative experimental results on image generation quality metrics. The authors claim that their approach:

Reduces dependency on traditional image augmentations and manual prompt engineering.
Produces "precise and contextually relevant prompts" that reflect the unique characteristics of each defect type.
Enables the generation of meaningful images by respecting anomaly constraints (e.g., valid defect co-occurrences).
Provides an extensible pipeline that can be integrated with existing text-to-image generative models to improve the quality of generated content for anomaly detection tasks.

Significance

The significance of this work lies in its shift from data-driven augmentation to knowledge-driven generation. By encoding expert domain knowledge into a Knowledge Graph, the approach addresses the limitations of manual prompt crafting and the lack of semantic understanding in standard augmentation techniques. The authors posit that this method leads to more balanced datasets and more effective training for anomaly detection models by ensuring that generated anomalies are not only visually plausible but also semantically and logically consistent with the specific industrial or product domain.