UniSpector: Towards Universal Open-set Defect Recognition via Spectral-Contrastive Visual Prompting

1. Problem Statement

Industrial visual inspection systems face a critical limitation: they are typically trained under a closed-set assumption, meaning they can only detect defect categories seen during training. In real-world manufacturing, new defect types emerge continuously, and the definition of "normal" shifts over time.

• Limitations of Existing Approaches:
  • Closed-set detectors: Fail on novel defects and require costly retraining and data rebuilding.
  • Anomaly detectors: Can identify that something is wrong but cannot distinguish between specific defect types (e.g., a scratch vs. a dent).
  • Text-driven open-set detectors: Struggle in industrial contexts because defects are often visually subtle and difficult to describe accurately with text, even for experts.
  • Existing Visual Prompting methods: While promising, they suffer from prompt embedding collapse. Due to high intra-class variance (e.g., rotation, lighting) and subtle inter-class differences, the embedding space becomes unstructured, causing the model to fail in distinguishing between different defect types or generalizing to unseen ones.

2. Methodology: UniSpector

UniSpector is a novel visual prompting framework designed to shift focus from naive "prompt-to-region matching" to the principled design of a semantically structured and transferable prompt topology. It consists of three core components:

A. Spatial-Spectral Prompt Encoder (SSPE)

To address high intra-class variance and subtle texture differences, SSPE redefines the visual prompt as a dual-domain descriptor:

• Spectral Domain: Computes the 2D Discrete Fourier Transform (DFT) of the prompt region. It extracts orientation-invariant radial frequency cues by averaging magnitude responses along concentric frequency rings. This suppresses variance caused by rotational transformations and captures fine-grained textures often missed in the pixel domain.
• Spatial Domain: Extracts spatial features using masked cross-attention guided by the Region of Interest (RoI) mask.
• Fusion: These two modalities are fused via lightweight MLPs to create a robust prompt embedding that is invariant to rotation but sensitive to subtle texture differences.

B. Contrastive Prompt Encoder (CPE)

To prevent embedding collapse, CPE explicitly regularizes the prompt embedding space into a semantically organized angular manifold:

• Mechanism: It employs an angular-margin contrastive loss (inspired by ArcFace).
• Objectives:
  1. Intra-class Compactness: Forces embeddings of the same defect class to cluster tightly, ensuring invariance to geometric variations.
  2. Semantic Angular Topology: Enforces a minimum angular margin between different classes. This structures the space so that semantically related defects are closer, while dissimilar ones are pushed apart.
• Result: This allows unseen defects to be placed semantically consistently in the embedding space rather than being arbitrarily projected.

C. Prompt-guided Query Selection (PQS)

Standard DETR-based models use learnable object queries that encode biases from training classes, limiting open-set generalization.

• Innovation: PQS dynamically selects a subset of vision tokens as object queries, conditioned on the visual prompt.
• Implementation: It computes cosine similarities between vision tokens and class prototypes, then uses a differentiable Gumbel-Softmax selection mechanism. This allows the model to learn a prompt-conditioned query distribution end-to-end, ensuring queries are adaptively initialized to the characteristics of the prompt exemplar.

3. Key Contributions

1. Identification of Structural Flaws: The authors identify that treating prompts as implicit representations leads to embedding collapse in industrial settings due to high variance and subtle differences. They propose that the prompt embedding space requires explicit structural regularization.
2. UniSpector Framework: A new architecture combining SSPE (for orientation-invariant, fine-grained feature extraction) and CPE (for structuring the embedding manifold) to enable robust open-set transfer.
3. Inspect Anything (InsA) Benchmark: The introduction of the first benchmark specifically for visual-prompt-based open-set defect localization. It includes:
   • In-domain evaluation: Testing on unseen defect types within the same data distribution.
   • Cross-domain evaluation: Testing on entirely different materials, imaging conditions, and defect morphologies.
   • Prompt-level generalization: Aggregating results across multiple random prompt seeds to ensure stability.

4. Experimental Results

The authors evaluated UniSpector against state-of-the-art visual grounding (e.g., GroundingDINO, YOLO-World) and visual prompting (e.g., DINOv, T-Rex2, SegGPT) baselines on the InsA benchmark.

• Performance Gains: UniSpector significantly outperforms all baselines.
  • In-domain: Achieves a 19.7% higher AP50b (detection) and 15.8% higher AP50m (segmentation) compared to the best baselines.
  • Cross-domain: Demonstrates superior transferability to unseen domains (e.g., 3CAD, VISION, VisA) where other methods often fail (near-zero performance for text-based models).
• Ablation Studies:
  • Removing SSPE or CPE leads to significant performance drops, confirming the necessity of spectral features and contrastive regularization.
  • PQS outperforms both fixed learnable queries and heuristic Top-K selection.
• Embedding Visualization: 3D PCA projections show that UniSpector forms compact clusters for seen classes with clear angular separation, whereas baselines (like DINOv) show scattered, overlapping embeddings.
• Closed-Set Competitiveness: Even in a fully supervised closed-set setting, UniSpector matches the performance of strong closed-set detectors (e.g., Mask2Former), proving it does not sacrifice accuracy for open-set capabilities.

5. Significance and Impact

• Scalable Industrial Inspection: UniSpector establishes a retraining-free paradigm. Manufacturers can introduce new defect types simply by providing visual prompts (exemplars) without collecting massive datasets or retraining the entire model.
• Robustness to Variance: By leveraging spectral cues and angular margins, the method effectively handles the high intra-class variance (rotation, lighting) and low inter-class separability typical of industrial defects.
• General Insight: The paper provides critical architectural insights for generic visual prompting, demonstrating that the structure of the embedding space is as important as the feature extraction itself for open-set recognition.
• Practicality: The method is data-efficient, requiring only 1–3 prompt images per class to stabilize performance, making it highly suitable for industrial environments where high-quality exemplar collection is limited.