💻 computer science

PADFormer: Pose-agnostic Anomaly Detection from Sparse View Images

PADFormer is a novel, efficient image-space approach for pose-agnostic anomaly detection that utilizes a Vision Transformer to reconstruct anomaly-free images from sparse views via cross-view masked reconstruction and ensemble inference, achieving state-of-the-art performance without the need for computationally expensive 3D reconstruction.

Original authors: Ruiqi Wang, Yiming Qian, Fenggen Yu, Yuxuan Lu, Dakuo Wang, Hao Zhang, Jing Huang

Published 2026-08-06

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Ruiqi Wang, Yiming Qian, Fenggen Yu, Yuxuan Lu, Dakuo Wang, Hao Zhang, Jing Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a quality inspector at a factory, but instead of standing at a fixed station, you are floating around a giant, spinning sculpture made of thousands of tiny parts. Your job is to spot a single scratch or a missing bolt. The problem? The sculpture is turning, and sometimes it's upside down, sideways, or hiding parts of itself behind other pieces. In the world of computer vision, this is the challenge of Anomaly Detection: teaching a computer to find defects using only pictures of "perfect" items. Usually, computers are like rigid robots; they expect the object to be in the exact same pose every time. If you show them a perfect cup from the front, they can spot a crack on the front. But if you show them a cup with a crack on the side, taken from a weird angle, the robot gets confused because it has never seen a cup from that angle before. This is where Pose-Agnostic Anomaly Detection comes in—a fancy way of saying, "Can the computer find the broken part no matter how the object is twisted or turned?"

For a long time, the only way to solve this was to build a full 3D hologram of the object. Imagine taking hundreds of photos of a toy from every possible angle, feeding them into a super-computer to build a virtual 3D model, and then rotating that model to match the new photo to see what's wrong. It works, but it's like trying to build a house out of sandcastles just to find a pebble; it takes forever, needs a mountain of data, and is incredibly expensive. The researchers behind this paper asked a simpler question: What if we didn't need the 3D model at all? What if we could just "imagine" what the perfect object looks like from this weird angle, right in the 2D picture?

Enter PADFormer, a new method that acts like a super-powered art restorer. Instead of building a 3D model, PADFormer looks at a few photos of a perfect object (say, 2 to 10 images) and a new photo of the object that might be broken. It then uses a special type of AI called a Vision Transformer (think of it as a brain that looks at an image in tiny puzzle pieces) to play a game of "fill in the blanks." The computer takes the new photo, covers up random patches of it (like putting sticky notes over parts of a drawing), and tries to redraw those missing parts using only the "perfect" photos it has seen. Because it was trained only on perfect objects, it doesn't know how to draw the broken part. So, when it tries to fill in the sticky note, it draws the "perfect" version of that spot. If the original photo had a scratch or a missing piece, the computer's drawing will look different from the real photo. By comparing the two, the computer instantly spots the defect, even if the object is twisted in a way it has never seen before.

The paper demonstrates that this approach is a game-changer for efficiency. While older methods required hundreds of photos and complex 3D reconstruction to work, PADFormer achieves state-of-the-art results using just a handful of reference images (as few as 2). In tests on datasets like MAD-SIM and PIAD, PADFormer significantly outperformed previous methods. For instance, in a "2-shot" scenario (using only two reference images), it achieved an image-level accuracy score of 78.8, compared to the next best method's 57.9. It also managed to locate defects on a pixel-by-pixel level with a score of 89.2, beating the competition by a wide margin. The authors show that this works not just for weird angles, but also for standard tasks where the object is in a fixed position, proving that their method is versatile.

The secret sauce lies in how PADFormer handles the "puzzle pieces." It doesn't just look at the whole picture; it uses a Dynamic Patch Selection mechanism. Imagine you are trying to fix a torn map. Instead of looking at the whole map, you find the specific, matching corner of a different map that fits the tear perfectly, even if that map is rotated. PADFormer does this mathematically, aligning the "perfect" reference patches to the "broken" query patches before trying to fill them in. It also uses a special "memory bank" of Anomaly-Agnostic Tokens—tiny, learned hints about what "normal" looks like globally and locally—to help it guess the right shape even when the object is upside down.

Crucially, the paper rules out the need for 3D reconstruction entirely. The authors argue that the heavy lifting of building 3D models is unnecessary and often impractical for real-world factories where you can't take hundreds of photos of every single item. They also show that while other methods might work well if the object is perfectly aligned, they fail miserably when the viewpoint changes. PADFormer, however, is designed specifically for this chaos. It doesn't just guess; it learns to reconstruct the "ideal" version of the image directly in 2D space.

In their experiments, the researchers found that PADFormer is not only accurate but also fast. By running the reconstruction process multiple times (specifically 15 passes with different random masks) and averaging the results, the system ensures it covers every inch of the image. They also discovered that measuring the difference between the original and the reconstructed image in a specific color space called CIELAB (which mimics how human eyes see color) was more effective than using standard RGB colors. This allowed them to catch subtle defects like tiny stains or burrs that other methods missed.

The paper concludes that PADFormer bridges the gap between high-tech 3D methods and simple 2D image processing. It suggests that we don't need to build a virtual world to find a broken toy; we just need a smart enough artist who knows what a perfect toy looks like from every angle. While the method has limitations—such as needing multiple passes to get a complete picture and potentially struggling with very large anomalies—it represents a significant step forward. It proves that with the right "imagination," computers can spot the weird and the broken, no matter how the world spins around them.

Technical Summary: PADFormer

Problem Statement

Pose-agnostic Anomaly Detection (PAD) aims to identify shape or texture anomalies in objects captured from arbitrary, unknown viewpoints. While traditional anomaly detection methods assume fixed, predefined viewpoints, real-world industrial inspection often involves significant pose variations and occlusions. Existing PAD solutions typically rely on complex 3D reconstruction pipelines (e.g., Neural Radiance Fields or 3D Gaussian Splatting) that require dense multi-view training data (often 200+ images) with ground-truth camera poses. These requirements create a critical bottleneck for real-world deployment, where obtaining comprehensive multi-view coverage with precise pose annotations is often infeasible or prohibitively expensive. Furthermore, standard Few-Shot Anomaly Detection (FSAD) methods assume geometric alignment between query and reference images, rendering them ineffective when viewpoints differ significantly.

Methodology: PADFormer

PADFormer proposes a novel image-space approach that bypasses explicit 3D reconstruction. It leverages Vision Transformers (ViT) to directly reconstruct anomaly-free versions of query images using only a sparse set of reference images (e.g., 2–10 views), regardless of the pose differences between them.

Core Architecture

The model employs an asymmetric encoder-decoder transformer architecture:

Encoder: Both the query image (with random masking) and $N$ anomaly-free reference images are tokenized and processed through weight-sharing ViT encoders. The query image undergoes asymmetric masking (40% of patches hidden), forcing the model to learn to reconstruct missing regions using information from the intact reference views.
Dynamic Patch Selection with Spatial Alignment: To handle pose variations efficiently, PADFormer introduces a Spatial Transformation Network (STN). The STN aligns reference feature grids to the query feature grid via learned affine transformations before similarity computation. A Dynamic Patch Selection mechanism then identifies the top- $k$ most relevant reference patches for each query position based on a hybrid similarity score (combining feature cosine similarity and spatial proximity). This reduces computational overhead while ensuring the model attends to pose-aligned evidence.
Cross-Attention Decoder: The decoder reconstructs masked query patches using 8 ViT layers. It integrates three sources of information via cross-attention:
- Query Context: Visible patches from the query image.
- Reference Evidence: The selected, pose-aligned reference patches.
- Anomaly-Agnostic Tokens: Learnable global and local tokens that encode normal appearance priors independent of specific anomaly types. These tokens allow the model to suppress anomalous patterns during reconstruction, effectively "hallucinating" the normal version of the object.

Training Strategy

PADFormer utilizes a two-stage training process exclusively on anomaly-free data:

Within-View Training: The model learns geometric invariance by reconstructing a query image from self-augmented versions of itself (rotations and deformations).
Cross-View Training: A unified model is trained across all object categories, learning to reconstruct query images using reference images from different views of the same object category. This enables generalization to unseen viewpoints and object instances.

Inference Procedure

During inference, the model performs multiple forward passes ( $I=15$ ) with different random masking patterns to ensure comprehensive spatial coverage.

Ensemble Reconstruction: Multiple anomaly-free reconstructions are generated.
Error Mapping: Pixel-wise differences between the query and reconstructions are computed in CIELAB color space (after Gaussian filtering to reduce noise).
Consistency Filtering: To distinguish genuine anomalies from consistent reconstruction artifacts (e.g., batch variations), the method computes the variance of error maps across the ensemble. Regions with low variance (consistent reconstruction) are suppressed, while high-variance regions (inconsistent reconstruction indicative of anomalies) are retained.

Key Contributions

Sparse-View PAD Formulation: The paper formalizes the sparse-view PAD setting, addressing the gap between research assumptions (dense multi-view data) and real-world constraints where only a few reference images are available.
3D-Free Reconstruction: PADFormer eliminates the need for computationally expensive 3D reconstruction and test-time pose optimization. It achieves pose consistency through learned masked reconstruction in 2D image space.
Novel Architectural Components: The integration of STN for pose alignment, dynamic patch selection for efficiency, and anomaly-agnostic tokens for robust prior learning enables effective learning from sparse views under significant pose variations.
Dual Capability: The method achieves state-of-the-art (SOTA) performance on PAD benchmarks while maintaining competitive performance on classic Few-Shot Anomaly Detection (FSAD) tasks, demonstrating versatility across different geometric constraints.

Experimental Results

The authors evaluated PADFormer on the MAD-SIM and PIAD datasets for PAD, and MVTec-AD and ViSA for FSAD.

PAD Performance: In few-shot settings (2-shot, 4-shot, 10-shot), PADFormer significantly outperforms existing 3D-based methods (OmniAD, SplatPose, PIAD) and 2D-based methods (PromptAD, UniVAD).
- On the MAD-SIM dataset (4-shot), PADFormer achieved 81.3% image-level AUROC and 92.9% pixel-level AUROC, surpassing the second-best method by over 20% in image-level AUROC.
- In the All-shot setting, PADFormer achieved the best image-level AUROC (97.8% on MAD-SIM) despite being trained jointly on all categories, whereas competitors often require per-object training.
FSAD Performance: On standard FSAD benchmarks (MVTec-AD, ViSA), PADFormer achieved competitive results (e.g., 94.7% image-AUROC on MVTec-AD with 4-shot), comparable to specialized FSAD methods like UniVAD, despite PADFormer being designed for the more challenging pose-agnostic setting.
Efficiency: The method avoids the heavy computational costs associated with 3D reconstruction and test-time pose optimization, offering superior inference efficiency.

Significance and Claims

The paper claims that PADFormer represents a significant step forward by making pose-agnostic anomaly detection practical for scenarios with limited data. By demonstrating that high-quality anomaly detection can be achieved without 3D reconstruction or dense multi-view datasets, the method addresses a critical barrier to industrial deployment. The authors emphasize that their approach successfully bridges the limitations of both traditional FSAD (which fails on pose variations) and existing PAD methods (which require impractical data density). The results suggest that learned masked reconstruction in 2D space, guided by pose-aligned reference selection and anomaly-agnostic priors, is a robust and efficient alternative to geometric 3D modeling for industrial inspection.