Technical Summary: SteelDefectX

Problem Statement

Steel surface defect analysis is a critical component of industrial quality control. While deep visual models have achieved success in industrial inspection, existing benchmarks for steel defects (e.g., NEU, GC10, X-SDD, S3D) remain limited to coarse-grained annotations, typically providing only class labels or numerical bounding boxes. These datasets lack the rich semantic descriptions necessary to capture the visual characteristics, spatial distribution, and potential causes of defects. Consequently, current benchmarks are ill-suited for evaluating emerging vision-language models (VLMs), which rely on expressive and structured textual supervision to achieve fine-grained understanding and generalization. Furthermore, existing attempts to apply vision-language paradigms to industrial defects often rely on simple, template-based descriptions derived solely from class labels, failing to capture the high variability and intrinsic complexity of real-world steel defects.

Methodology

To address these gaps, the authors introduce SteelDefectX, a multi-form vision-language dataset and benchmark. The methodology encompasses dataset construction, a multi-level annotation framework, and a comprehensive evaluation benchmark.

1. Dataset Construction

SteelDefectX unifies four widely used public datasets (NEU, GC10, X-SDD, and S3D) along with selected samples from FSC-20 and ESDIs-SOD. The integration process involved:

• Standardization: Resizing all images to 256 × 256 and filtering low-quality samples.
• Taxonomy Unification: Cross-verifying and merging visually and semantically similar subclasses into a consistent taxonomy of 25 defect categories.
• Scale: The final dataset comprises 7,778 images with pixel-level segmentation masks.

2. Multi-Form Annotation Framework

The core innovation is the provision of multi-level textual annotations that bridge low-level visual observations and high-level semantic reasoning. These annotations are categorized into class-level and sample-level forms:

• Class-Level (T1): Each of the 25 categories is described with a name, representative visual attributes (shape, texture, color), and potential industrial causes (e.g., rolling defects, material inclusions).
• Sample-Level: Each image is annotated with three complementary textual forms:
  • T2 (Natural Language Descriptions): Free-form, descriptive sentences generated via a multi-stage pipeline involving Large Language Models (LLMs), semantic-similarity filtering, dimension coverage evaluation, and manual refinement. These descriptions cover six dimensions: appearance, shape, size, depth, position, and contrast.
  • T3 (Structured Attributes): A nine-field tuple encoding fine-grained characteristics (Defect type, Shape, Direction, Spatial Distribution, Number of Defects, Position, Scale, Polarity, Saliency). Attributes are derived through a hybrid process: rule-based computation from segmentation masks (e.g., position, scale, polarity) and constrained semantic inference using LLMs for attributes like shape and direction.
  • T4 (Template-Based Sentences): Structured attributes (T3) are linearized into standardized sentences to provide a controllable, intermediate text format.

3. Benchmark and Evaluation

The authors established a unified benchmark covering multiple vision-language tasks:

• Vision-Language Classification: Evaluating models (CLIP, OpenCLIP, Long-CLIP, etc.) on matching images to text prompts.
• Vision-Language Segmentation: Assessing pixel-level localization using text embeddings.
• Cross-Dataset Transfer: Testing generalization on unseen datasets (Aluminum surface defects, Seamless steel tube defects).
• Additional Analyses: Structured attribute ablation, vision-language retrieval, text-guided localization, and image-text similarity analysis.

Key Results

Experimental results reveal a distinct trade-off between the structure and flexibility of textual representations:

• Classification: Structured attributes (T3) consistently provided the most stable and effective supervision, achieving the highest accuracy (e.g., 94.45% Acc with Long-CLIP ViT-L/14 under T3-T0). This suggests that explicit, less ambiguous semantic signals benefit classification tasks.
• Cross-Dataset Transfer: Free-form natural language descriptions (T2) demonstrated superior transferability to unseen datasets compared to structured or template-based text. This indicates that semantic richness and linguistic flexibility are crucial for generalization.
• Segmentation and Grounding:
  • In class-level aggregation for segmentation, richer descriptions (T2-T4) did not consistently outperform simple class names (T0), suggesting that fine-grained information may be diluted when aggregated.
  • However, in text-guided localization (1-to-1 matching), T2 achieved the best AUROC and F1-max, while T3 yielded the highest IoU. This highlights that free-form descriptions improve fine-grained spatial grounding, whereas structured attributes offer strong global semantic alignment.
• Retrieval: Structured representations (T3, T4) outperformed free-form descriptions (T2) in image-text retrieval tasks, indicating that structured signals facilitate more stable cross-modal alignment.

Significance and Claims

The paper claims that SteelDefectX provides a new benchmark for studying semantic alignment and generalization in industrial vision-language learning. Its primary contributions are:

1. First Multi-Form Dataset: It is the first steel surface defect dataset to provide multi-form textual annotations (class-level, free-form, structured, and template-based) alongside pixel-level masks.
2. Systematic Analysis of Text Design: By offering diverse annotation forms, the dataset enables a systematic investigation into how textual structure influences model behavior. The findings suggest that different tasks require different textual strategies: structured attributes for stable classification and retrieval, and free-form descriptions for transferability and fine-grained grounding.
3. Industrial Relevance: The dataset aims to facilitate the development of more interpretable and generalizable inspection systems, potentially aiding in zero-shot and few-shot recognition, text-guided generation, and industrial multimodal reasoning.

The authors conclude that while structured annotations facilitate stable alignment, free-form natural language descriptions are critical for transferability and spatial grounding, highlighting the critical role of textual design in industrial vision-language learning.