Unification of Closed-Open Industrial Detection Scenarios: New Large-Scale Benchmarks,Challenges and Baselines

Technical Summary: Unification of Closed-Open Industrial Detection Scenarios

Problem Statement
Despite the remarkable success of Large-scale Visual-Language Models (LVLMs) in natural visual tasks, their application to industrial defect detection faces two fundamental limitations. First, there is a scarcity of large-scale industrial datasets that cover diverse defect categories across multiple domains; existing datasets are typically confined to single fields (e.g., PCBs or textiles) and operate in closed-set scenarios, limiting generalization to open environments. Second, current LVLMs rely heavily on manual prompts (points, boxes, masks) to segment objects. In complex industrial scenes characterized by noise and subtle defects, these manual inputs introduce subjective noise and lack the text-visual interaction necessary for fine-grained understanding. Furthermore, simply transferring knowledge from natural scenes to industrial domains fails to address significant feature distribution shifts.

Methodology
To address these challenges, the authors propose a unified framework comprising a new benchmark and a novel network architecture.

1. MMIOC-1M Benchmark:
   The authors introduce the Large-Scale Multi-Modal Industrial Open-Closed benchmark (MMIOC-1M). This dataset contains over one million samples derived from 31 different industrial defect fields. It is structured hierarchically with 14 super-categories, 29 industrial scenes, and 351 defect subcategories. Unlike previous datasets, MMIOC-1M supports both open-vocabulary and closed-set detection tasks. The construction involved a three-stage process: defining defect attributes via expert screening and GPT-4V semantic retrieval, image screening and calibration using adversarial annotation methods (GroundingDino and YOLOv8) combined with expert verification, and the division of open-closed subsets based on semantic similarity in embedding spaces.

2. RTVPNet (Refined Text-Visual Prompt Network):
   The authors propose RTVPNet, a network designed to enhance LVLM capabilities in industrial scenarios through three key innovations:

   • Expert-Assisted Domain Projection: To bridge the gap between general vision models and industrial domains, the network employs a two-stream architecture. A pre-trained expert model (e.g., C2F) provides supervision to a Mobile-SAM backbone via a domain-aware projection module. This involves masking and reconstructing features to inject industrial knowledge, with the expert model dynamically updated during training to ensure continuous adaptation.
   • Energy-Based Sparse Sampling (Refined Visual Prompts): To eliminate the need for manual prompts and reduce background noise, the network generates visual prompts automatically. It calculates an uncertainty score for each pixel based on an energy function derived from coarse-grained segmentation features. A sparse sampling strategy then selects high-frequency pixels (those with values greater than the mean in local patches) to form refined visual prompts, focusing on object contours while suppressing irrelevant background.
   • Bidirectional Text-Visual Interaction: The network implements a mutual interaction mechanism. In the Text-to-Visual direction, text embeddings guide multi-scale visual features via a Sigmoid attention mechanism to focus on defect locations. In the Visual-to-Text direction, pixel-level similarity scores between image and text features are used to enhance text embeddings with spatial details, compensating for the lack of visual information in simple text descriptions.

Key Contributions

• MMIOC-1M Benchmark: The first large-scale, unified benchmark for industrial open-closed detection, offering comprehensive multi-domain coverage (1M+ samples, 351 subcategories) and standardized evaluation protocols for visual grounding, object detection, and visual question answering.
• RTVPNet Baseline: A novel architecture tailored for industrial detection that integrates expert-assisted domain projection, automated refined visual prompt generation, and bidirectional cross-modal interaction.
• Comprehensive Evaluation: The establishment of standardized protocols and extensive experiments demonstrating the benchmark's difficulty and the proposed method's efficacy across MMIOC-1M, LVIS, and COCO datasets.

Experimental Results
Extensive experiments demonstrate that RTVPNet achieves state-of-the-art performance while maintaining computational efficiency:

• Open-Scene Performance: On MMIOC-1M, RTVPNet-S achieves an AP50 of 26.7%, outperforming YOLO-World-S by 2.7% and DOSOD-S by 3.4%, with significantly fewer parameters (76M) and GFLOPs (17). It also surpasses larger LVLMs (e.g., Qwen2.5-VL 3B) despite having approximately 1/40th the parameter count.
• Closed-Scene Performance: On MMIOC-1M closed scenes, RTVPNet-L achieves an AP of 41.3%, outperforming traditional expert models (like YOLOv12L) by approximately 15% and matching or exceeding large LVLMs like Qwen3-VL 4B.
• Generalization: On the LVIS dataset, RTVPNet improves AP by 3.2% over YOLO-World, validating its ability to generalize to open-vocabulary scenarios. On COCO, it improves multi-scale AP by approximately 1% over the YOLOv11 baseline.
• Qualitative Analysis: t-SNE visualizations show that RTVPNet produces more compact and consistent feature clusters for invisible categories compared to other methods. Visualizations of detection results confirm superior robustness against background noise and scale variations in both open and closed scenarios.

Significance
The paper claims that MMIOC-1M fills a critical gap in industrial LVLM research by providing the first unified benchmark for open-closed scenarios, catalyzing the development of large-scale industrial models. The proposed RTVPNet offers a practical solution to the domain shift and prompt subjectivity problems, enabling LVLMs to adapt rapidly to industrial environments with high accuracy and low computational cost. The work suggests that the combination of expert-guided domain transfer and refined text-visual interaction is essential for overcoming the challenges of inter-class similarity, intra-class differences, and long-tail distributions inherent in industrial defect detection.