💻 computer science

Keep the Needle, Prune the Haystack: Defect-Preserving Token Pruning for Efficient Zero-Shot Anomaly Detection

The paper proposes KeepAD, a defect-preserving token pruning framework that strategically retains anomalous tokens while aggressively removing redundant normal tokens across different network depths, enabling efficient zero-shot anomaly detection with up to a 7.9× speedup and minimal performance degradation.

Original authors: Yanning Hou, Jingyuan Zhang, Xiaoyun Wang, Qixiang Ma, Sihang Zhou, Ke Xu

Published 2026-08-11

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Yanning Hou, Jingyuan Zhang, Xiaoyun Wang, Qixiang Ma, Sihang Zhou, Ke Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to find a single, tiny scratch on a brand-new, expensive car. You have a super-smart robot assistant with a camera that can see every single inch of the vehicle. But here's the catch: the robot is so thorough that it inspects every square millimeter of the car, even the parts that are perfectly clean and shiny. It spends 99% of its time checking the clean paint, just to be absolutely sure, before it finally spots the scratch. This is incredibly slow and wasteful, especially if you have to inspect thousands of cars a day.

This is exactly the problem facing modern "Zero-Shot Anomaly Detection." In the world of computer vision, "Zero-Shot" means a computer can spot defects in things it has never seen before—like finding a crack in a new type of medical scan or a flaw in a factory part it wasn't trained on. To do this, these computers use massive "Vision Transformers" (ViTs), which are like giant brains that break an image into thousands of tiny puzzle pieces (called "tokens") and analyze them all. The problem is that these brains are heavy and slow. They treat a perfectly normal image and a broken one the same way, crunching numbers on every single piece of the puzzle, even though the "bad" piece is usually just a tiny speck in a sea of "good" ones.

Enter KeepAD, a new method that acts like a smart, risk-aware editor for these giant brains. Instead of letting the robot check every single inch of the car, KeepAD teaches the robot to quickly scan the surface, keep an eye on the most suspicious spots, and ignore the boring, perfect parts. But it does this with a very careful rule: "Don't throw away the evidence." If the robot is unsure if a spot is a scratch or just a shadow, it keeps it. It only deletes the boring, safe parts. The result? The robot becomes incredibly fast—up to nearly 8 times faster in some tests—without missing the tiny defects it's supposed to find. It's like having a detective who can spot a needle in a haystack in a fraction of the time, simply by ignoring the hay that is definitely just hay.

The Problem: The "Haystack" Trap

In the world of industrial and medical safety, finding defects is a game of "find the needle in the haystack." Most of the time, the images are perfect (the hay), and the defects are rare and tiny (the needle). Current AI models are like a very diligent but slow librarian who reads every single page of every book in the library to find one typo. Even though the typo is only on page 42, the librarian reads pages 1 through 41 with the same intensity.

This is inefficient. The paper points out a specific danger called the "Asymmetric Pruning Risk." If you try to speed things up by deleting "unimportant" pages (tokens), you might accidentally throw away the page with the typo. In normal image recognition (like identifying a cat), deleting a few pixels doesn't matter much because the cat's face is everywhere. But in defect detection, the "cat" is the whole image, and the "defect" is a tiny, hidden detail. If you delete the wrong token, you lose the only evidence of the problem.

The Solution: KeepAD's Two-Stage Strategy

The authors of this paper propose KeepAD (Keep Anomaly Detection), a system that acts like a smart filter. It doesn't just delete things randomly; it uses a two-step process that changes its strategy as it "looks" deeper into the image.

Stage 1: The "Safety Net" (Shallow Layers)
When the AI first looks at the image, it's not yet sure what a defect looks like. It's like looking at a blurry photo. If it tries to guess which parts to delete now, it might accidentally delete the defect. So, KeepAD uses a Coverage-Preserving strategy. Imagine the image is a grid of 2x2 squares. KeepAD says, "No matter what, we must keep at least one piece from every single 2x2 square." This ensures that even if a defect is tiny and isolated, it won't be completely wiped out. It also adds a "Rescue" mechanism: if a spot looks slightly risky, it gets saved even if it's not the most obvious thing. This stage is conservative; it's better to keep a little extra hay than to lose the needle.

Stage 2: The "Sniper" (Deep Layers)
As the AI processes the image deeper, it starts to understand the difference between normal texture and a real defect. Now, it can be more aggressive. KeepAD switches to an Anomaly-Adaptive mode. It uses "prototypes"—mental templates of what "normal" looks like and what "abnormal" looks like. If a token clearly matches the "normal" template, it gets deleted. If it looks like a defect, it stays.

Crucially, this second stage is Image-Adaptive. It doesn't use a fixed rule for every picture. If an image looks very suspicious (lots of potential defects), KeepAD keeps more tokens to be safe. If the image looks very clean, it deletes more tokens to save time. It's like a security guard who checks a suspicious bag thoroughly but just glances at a clear, empty one.

The Secret Sauce: Learning Without Losing

One of the hardest parts of this system is teaching the AI to make these decisions. If the AI deletes a token, the computer can't "see" it anymore to learn from it. To solve this, the authors use a trick called Dense-to-Sparse Self-Distillation.

Imagine a teacher and a student. The "Teacher" is the full, slow AI that looks at every token. The "Student" is the fast, pruned AI that only looks at a few. During training, the Teacher looks at the whole image and says, "Hey, look at this spot! Even though it looked boring at first, it turned out to be important later." The Student learns to pay attention to those spots before it deletes anything. This allows the Student to become fast without needing to see the whole picture every time. Once the training is done, the Teacher is gone, and the Student runs the show alone, super fast.

The Results: Fast and Accurate

The researchers tested KeepAD on 13 different benchmarks, ranging from industrial parts (like metal sheets and circuit boards) to medical images (like brain scans and X-rays).

Speed: KeepAD is a speed demon. It reduced the number of tokens the AI had to process to less than 20% of the original amount. In the most aggressive setting, it was 7.9 times faster than the strongest existing method based on CLIP (a famous AI model).
Accuracy: Despite throwing away so much data, it didn't miss the defects. The drop in accuracy was tiny—less than 2.7 percentage points on average. In many cases, it was almost as accurate as the slow, full version.
Reliability: The system successfully avoided "complete misses" (where a defect is entirely deleted). The "Coverage-Preserving" stage in the beginning was key to this, ensuring that no tiny defect was ever lost in the shuffle.

Why It Matters

This paper doesn't just suggest a new way to make AI faster; it solves a specific, dangerous problem where speed usually comes at the cost of safety. By proving that you can prune (delete) most of the data without losing the critical "needle," KeepAD makes it possible to run high-level defect detection on real-world systems that need to be fast, like assembly lines or emergency medical screening. It shows that with the right strategy—being careful at the start and smart at the end—you can have your cake (speed) and eat it too (accuracy).

Technical Summary: KeepAD – Defect-Preserving Token Pruning for Efficient Zero-Shot Anomaly Detection

1. Problem Statement

Zero-shot visual anomaly detection (ZSAD) has advanced significantly using Vision-Language Models (VLMs) and Vision Transformers (ViTs). However, existing methods typically perform dense computation over all image patches (tokens) across every Transformer layer. This approach is inefficient because:

Data Sparsity: In real-world scenarios, anomalous images are rare, and within an anomalous image, the decisive evidence (defects) usually occupies only a small, localized region.
Asymmetric Pruning Risk: Unlike generic image classification where discarding non-dominant tokens is acceptable, anomaly detection faces a critical asymmetry. Retaining redundant normal tokens incurs computational waste, but removing an anomalous token—even a subtle one—can irreversibly erase the only evidence required for detection and localization.
The Early-Pruning Paradox: Pruning early layers offers the greatest computational savings, but shallow representations often lack the semantic reliability to distinguish subtle defects from normal texture variations. Conversely, deeper layers provide reliable anomaly cues but leave less computation to save.

Current token pruning methods for classification or generation are ill-suited for ZSAD because they prioritize dominant semantics, potentially discarding the sparse, inconspicuous tokens that constitute the anomaly.

2. Methodology: KeepAD

The authors propose KeepAD, a defect-preserving token pruning framework designed to navigate the asymmetric pruning risk and the early-pruning paradox. The method operates on a frozen CLIP ViT backbone (e.g., ViT-L/14@336px) and employs a two-stage training process followed by a single sparse inference pass.

2.1 Two-Stage Training

Stage 1 (Dense Detector Training): A lightweight anomaly head is trained on top of the frozen ViT backbone. This head learns normal and anomaly prototypes at intermediate layers. Crucially, it produces a "dense-to-sparse" teacher signal: an unpruned pass generates deep-layer anomaly priors that serve as supervision targets for the pruning selectors.
Stage 2 (Selector Training): The backbone and head are frozen. Two lightweight token selectors are inserted at specific layers (L8 and L12) and trained to route tokens. The detector itself is never updated.

2.2 Progressive Defect-Preserving Pruning

KeepAD adapts its pruning strategy based on network depth:

Early Stage (L8): Coverage-Preserving Selection
- Challenge: Anomaly semantics are unreliable; prototypes are not yet discriminative.
- Strategy: The selector avoids global top-k selection, which might concentrate tokens on dominant normal regions. Instead, it partitions the image grid into $2 \times 2$ blocks and guarantees that at least one representative token from every non-empty block is retained.
- Rescue Mechanism: Any remaining token budget is allocated deterministically to "rescue candidates" (e.g., high-scoring tokens from distinct grid cells) to prevent the total loss of small or isolated defects.
Deep Stage (L12): Anomaly-Adaptive Selection
- Challenge: Anomaly semantics are reliable, but computation must be aggressively reduced.
- Strategy: The selector combines visual salience with affinities to the frozen normal and anomaly prototypes learned in Stage 1.
- Image-Adaptive Budget: Instead of a fixed pruning ratio, the token budget is dynamically adjusted per image based on the distribution of anomaly evidence. Images with pronounced anomalies receive a higher token budget, while evidence-poor (likely normal) images are pruned more aggressively.
- Block-Level Pruning: To maintain spatial coherence, L12 selects entire blocks of tokens rather than isolated tokens.

2.3 Dense-to-Sparse Self-Distillation

To address the difficulty of training the shallow (L8) selector without direct gradient flow from the final detection loss, KeepAD utilizes self-distillation. The frozen, unpruned "teacher" branch provides deep-layer anomaly responses as explicit preservation targets for the shallow "student" selector. This guides the early router to retain tokens that are likely to become anomaly-relevant in deeper layers, without adding inference overhead.

2.4 Dense Recovery and Inference

During inference, only the hard-pruned tokens are processed. To restore dense localization:

Spatial Recovery: Dropped patch positions are mapped to their nearest surviving token on the original grid. The anomaly response of the survivor is copied to the dropped position.
Scoring Integrity: Crucially, image-level anomaly scores are aggregated only from physical survivors. Recovered positions are used solely for pixel-level map generation and do not influence the final image-level classification, preventing artificial evidence from inflating scores.

3. Key Contributions

Problem Formulation: The paper identifies and formalizes the "asymmetric pruning risk" and the "early-pruning paradox" specific to zero-shot anomaly detection, distinguishing it from standard token pruning tasks.
KeepAD Framework: A progressive routing mechanism that transitions from conservative, coverage-preserving spatial selection in shallow layers to aggressive, prototype-guided, image-adaptive pruning in deeper layers.
Training Strategy: The introduction of dense-to-sparse self-distillation to supervise shallow token selection using deep anomaly priors, enabling effective training without retraining the backbone.
Empirical Validation: Extensive evaluation across 13 benchmarks (6 industrial, 7 medical) demonstrating that KeepAD can reduce token retention to below 20% while maintaining competitive accuracy.

4. Experimental Results

Evaluated on a single NVIDIA RTX 5090 GPU, KeepAD demonstrates significant efficiency gains with minimal accuracy degradation:

Throughput: Achieves up to a 7.9× speedup over the strongest CLIP-based baseline (Bayes-PFL) and a 1.7× speedup over the strongest ViT-based baseline (VisualAD) at competitive accuracy.
Accuracy: At the most aggressive operating point (retaining ~14–18% of tokens), the average degradation in image-level (I-AUROC) and pixel-level (P-AUROC) performance is limited to within 2.7 percentage points.
Robustness: The method transfers effectively between industrial (MVTec-AD, VisA) and medical (OCT, MRI, Endoscopy) domains without domain-specific adaptation.
Ablation Insights:
- Removing L8 pruning causes a significant drop in speed (6.5–8.3 img/s), confirming early pruning as the primary efficiency lever.
- Replacing coverage-preserving selection with global top-k selection drastically increases the "Complete Miss Rate" (CMR), proving the necessity of local coverage constraints.
- Image-adaptive budgeting outperforms fixed or shuffled budgets, particularly in reducing CMR.

5. Significance and Claims

The paper claims that KeepAD successfully reconciles the conflicting goals of aggressive token reduction and sparse defect evidence preservation. By formulating token selection as a high-recall, anomaly-aware routing problem, KeepAD avoids the "complete miss" failures common in generic pruning methods.

The authors emphasize that their approach is not merely a speed optimization but a structural adaptation to the unique sparsity and risk profile of anomaly detection. The method achieves substantial inference acceleration (up to 7.9×) while maintaining detection and localization performance comparable to dense, non-pruned state-of-the-art models, making it a viable solution for practical, real-time zero-shot anomaly detection systems.