🤖 machine learning

When More References Hurt: Contamination-Aware DINOv2 Memory Banks for Few-Shot Steel Defect Detection

This paper proposes a contamination-aware memory bank construction method for few-shot steel defect detection that filters unverified industrial images to remove anomalous patches, thereby significantly improving detection performance compared to reference expansion or random removal strategies.

Original authors: Hannaneh Kalantari, Javad Khoramdel

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hannaneh Kalantari, Javad Khoramdel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast, humming factories where steel is rolled into thin, gleaming sheets, the surface must be flawless. Even a tiny scratch or a microscopic pit can ruin a batch of metal destined for cars or appliances. For decades, factories have relied on automated systems to catch these flaws, but teaching a computer to see them is a tricky business. Usually, engineers show the machine thousands of pictures of perfect steel and thousands more of steel with specific, labeled cracks and dents. The computer learns to spot the difference. However, in the real world, perfect labels are expensive and slow to make. Defects are rare, subtle, and constantly changing. This has led researchers to a different approach: instead of teaching the machine what a defect looks like, they teach it what "normal" looks like. The system is given a library of perfect examples, and if a new piece of steel doesn't match the library, it is flagged as broken. This method works beautifully when the library is perfect, but it faces a dangerous vulnerability: what if the library itself is contaminated with hidden flaws?

This is the precise problem tackled by a team of researchers at the University of Padua and the University of Vienna. They investigated a modern technique that uses a powerful, pre-trained artificial intelligence model to build this library of "normal" steel patches. The model breaks an image into thousands of tiny squares, or patches, and stores them in a memory bank. When a new image arrives, the system compares its patches against the bank. If a patch is too different from anything in the bank, it is marked as a defect. The danger lies in how the bank is built. In a factory, it is easy to collect hundreds of images of steel, but it is hard to verify that every single pixel in every image is truly perfect. If a researcher adds a hundred unverified images to the bank, hoping to give the system more variety, they might accidentally include images with hidden defects. The system, trusting its new library, would then learn that these defects are actually normal, and it would stop flagging them. The researchers asked a critical question: can we safely use these unverified images to expand our library without letting the bad patches in?

To answer this, the team turned to a dataset of steel strip images containing four types of defects. They started with a small, trusted set of images that were guaranteed to be defect-free. This small set formed a "seed" memory. They then took a larger set of unverified images—some of which contained hidden defects—and tried to mine them for useful, normal patches. Their first attempt was simple: they added every patch from the unverified images to the seed. The result was a disaster. Even though most of the steel in those images looked fine, the few hidden defects were enough to confuse the system. When they measured the system's ability to find flaws, the performance dropped significantly. In fact, they found that replacing just a tiny fraction of the memory bank with bad patches—less than one percent—caused the system's accuracy to plummet by more than a quarter. This proved that the purity of the reference library is the most important factor; a larger library full of hidden errors is worse than a smaller, clean one.

The researchers then developed a method to filter the unverified images before they ever touched the memory bank. They used the trusted seed images as a yardstick. Every patch from the unverified images was compared to the seed. If a patch looked very different from the trusted normal examples, it was likely a defect and was discarded. They found that by simply rejecting the twenty percent of patches that were most different from the trusted seed, they could remove the vast majority of the hidden flaws. This process, which they called distance-trimmed purification, was remarkably effective. It rejected nearly eighty percent of the anomalous patches hidden in the unverified images while keeping the vast majority of the good ones. The remaining patches were then merged with the seed and compressed into a final memory bank of a fixed size, ensuring a fair comparison with other methods.

The results showed that this careful filtering made all the difference. When the researchers compared their filtered bank to a bank built by simply adding all the unverified images, the filtered version was far superior at spotting defects. It also outperformed a bank built by randomly removing patches, proving that the improvement came from the smart filtering, not just from having fewer images. However, the researchers were careful to note that this method is a tool for when clean images are scarce. On the held-out data used for final testing, a bank built from eight completely clean images remained the strongest performer. The proposed method is designed for the difficult, real-world scenario where engineers have a small trusted core but a mountain of unverified data they want to use, offering a way to recover useful normal appearance from risky imagery when verified references are too few to build a large library on their own.

Ultimately, the study demonstrates that in the world of automated inspection, more data is not always better. Adding unverified images without a way to check them can teach the system to ignore real problems. The researchers showed that by using a small, trusted group of examples to vet the larger group, it is possible to safely expand the system's knowledge. They found that a simple rule—discarding the patches that look the most unlike the trusted normal ones—can strip away the contamination and recover the useful information hidden in risky data. This approach allows factories to leverage the abundance of cheap, unverified images they collect every day, turning a potential liability into a reliable asset, provided they are willing to filter out the noise before it enters the system.

Technical Summary: When More References Hurt: Contamination-Aware DINOv2 Memory Banks for Few-Shot Steel Defect Detection

Problem Statement
In industrial anomaly detection, particularly for steel surface defects, patch-memory detectors (e.g., AnomalyDINO) rely on a reference bank of "normal" image patches to define the boundary of normality. While these methods perform well with curated, defect-free training sets (like MVTec AD), real-world deployment often involves collecting large volumes of unverified industrial images. These images may contain subtle, unannotated defects. The core problem addressed is that unverified references introduce anomalous patches into the memory bank. Since nearest-neighbor detectors cannot distinguish between a mislabeled anomalous patch and a true normal patch once stored, even a small fraction of contamination can degrade performance significantly. The paper investigates whether a few trusted normal images can be used to safely extract useful normal patches from a larger set of unverified images without requiring defect masks.

Methodology: DT20
The authors propose DT20 (Distance-Trimmed 20%), a mask-free expansion strategy built upon the AnomalyDINO framework using frozen DINOv2 features. The method operates as follows:

Seed Bank Construction: A small set of verified, defect-free images (e.g., two images) forms a "clean seed memory" ( $M_c$ ).
Candidate Scoring: A larger set of unverified images provides candidate patches. Each candidate patch $u$ is scored based on its Euclidean distance to the nearest neighbor in the clean seed bank: $d(u) = \min_{m \in M_c} \|f(u) - m\|_2$ .
Distance Trimming: Candidates are ranked by this distance. The top 20% of patches with the largest distances (most dissimilar to the trusted seed) are rejected. This step assumes that patches far from the known normal distribution are likely to contain defects or significant anomalies.
Memory Consolidation: The retained 80% of candidates are merged with the seed bank.
Budget Enforcement: The combined set is reduced to a fixed memory budget (51,200 patches) using greedy coreset selection to ensure the final memory size matches that of baseline methods.

Key Contributions

Quantification of Contamination Sensitivity: The study demonstrates that patch-memory detectors are highly sensitive to reference contamination. Injecting just 0.5% anomalous patches into a clean bank reduces the Area Under the Precision-Recall Curve (AUPRC) by 26.3% (from 0.1030 to 0.0759).
Purification Strategy: The introduction of a distance-based trimming mechanism that rejects suspicious patches without requiring ground-truth defect masks during the training phase.
Rigorous Controls: The authors separate the effects of purification from memory size and reference count by comparing DT20 against:
- Unfiltered Expansion: Adding all unverified patches.
- Random Removal: Removing 20% of patches uniformly at random.
- Oracle: Removing patches overlapping ground-truth defects (upper bound).
- Exact-Budget Controls: Ensuring all methods use the same final memory size.

Results
Experiments were conducted on the Severstal Steel Defect Detection dataset using a DINOv2 ViT-S/14 backbone.

Contamination Rejection: In a setting with two clean seed images and eight unverified images, unfiltered expansion resulted in 9.46% anomalous patches in the memory. DT20 rejected 78.1% of these anomalous candidates, reducing residual contamination to 2.59%.
Performance Gains:
- On the development fold (Fold 0), DT20 achieved an AUPRC of 0.1084, outperforming unfiltered expansion (0.0950) and random removal (0.0952).
- On held-out folds (1 and 2), Clean-8 (8 verified clean images) remained the strongest strategy with a mean AUPRC of 0.1406, outperforming DT20 (0.1062). However, DT20 consistently outperformed unfiltered expansion across all five available fold-seed pairs, with a mean AUPRC gain of 0.0142.
Trade-offs: The method intentionally sacrifices some normal diversity (retaining 86.1% of normal candidates) to ensure safety, resulting in a reject precision of 0.369. However, this trade-off yields a net positive impact on detection performance compared to unfiltered expansion.

Significance and Claims
The paper concludes that reference purity is a first-order design variable in patch-memory anomaly detection. The primary significance lies in demonstrating that "more data" is not inherently better; unverified images are only useful if their contribution is explicitly filtered.

Conditional Utility: The authors claim that verified clean references remain the safest and most effective strategy when available, as evidenced by Clean-8's superior performance on held-out folds. Contamination-aware expansion (DT20) is presented as a viable alternative only when the trusted seed is small but additional unverified imagery is plentiful and exhaustive verification is too costly.
Deployment Rule: The proposed approach is not a replacement for acquiring certified normal references. Instead, it offers a mechanism to recover useful normal appearance variation from risky imagery when a clean-only bank cannot be constructed.
Limitations: The authors explicitly note that the held-out results are based on an incomplete cross-validation matrix (only folds 1 and 2 available for some methods) and that the 20% trim rate was selected on a single development fold. They also acknowledge that fixed-threshold F1 scores remain unstable, suggesting production systems would require explicit operating-point calibration.

The paper does not claim to solve the problem of defect localization without any supervision or to generalize to all industrial domains, but rather provides a specific, modest solution for expanding memory banks in few-shot steel defect detection scenarios where reference contamination is a known risk.

Technical Summary: When More References Hurt: Contamination-Aware DINOv2 Memory Banks for Few-Shot Steel Defect Detection

More like this