Boxes2Pixels: Learning Defect Segmentation from Noisy SAM Masks

1. Problem Statement

In industrial inspection (e.g., wind turbine blades), accurate pixel-level defect segmentation is crucial for predictive maintenance. However, obtaining dense pixel-level annotations is labor-intensive, costly, and requires domain expertise.

• Current Workaround: Using inexpensive bounding box annotations combined with foundation models like the Segment Anything Model (SAM) to generate pseudo-masks.
• The Challenge: SAM-generated pseudo-masks on industrial surfaces are systematically noisy. They often:
  • Hallucinate: Create false positives by segmenting background structures (shadows, textures) as defects.
  • Miss Defects: Fail to detect sparse, thin, or low-contrast defects (false negatives) even when enclosed by a bounding box.
• The Gap: Standard segmentation models trained on these noisy pseudo-labels tend to overfit the teacher's errors, inheriting its failure modes. Existing weakly supervised methods often assume the target occupies a significant portion of the box, which is not true for sparse industrial defects.

2. Methodology: Boxes2Pixels

The authors propose Boxes2Pixels, a data-centric distillation framework that treats SAM as a noisy teacher rather than a source of ground truth. The goal is to train a compact "student" model that is robust to systematic pseudo-label noise.

A. Data Pipeline

1. Input: Bounding box annotations (YOLO format) are provided.
2. Pseudo-Label Generation: SAM is run offline using these boxes as prompts to generate initial binary pseudo-masks.
3. Training: A lightweight student network is trained using these noisy masks. Crucially, SAM is not required at inference time.

B. Hierarchical Student Architecture

The student network is designed to balance semantic stability with high-frequency spatial detail:

1. Global Semantic Branch: Utilizes frozen DINOv2 ViT-S/14 features.
   • Only bias and normalization parameters are updated via BitFit adaptation to prevent overfitting to noise.
   • Intermediate features from layers {1, 2, 4, 7} are fused top-down using residual blocks to ensure semantic stability against high-frequency artifacts.
2. Local Detail Branch: A lightweight CNN branch processes the raw RGB input directly.
   • Extracts fine-grained spatial features to capture thin structures (e.g., hairline cracks) that transformer patch representations might miss.
3. Feature Fusion: The global and local features are concatenated and processed by a Feature Mixer.
4. Dual Prediction Heads:
   • Binary Defect Head: Predicts foreground (defect) vs. background.
   • Fine-Grained Semantic Head: Predicts specific defect classes (e.g., Damage, Dirt).

C. Training Objectives & Self-Correction

The loss function is designed to handle noise and sparse defects:

1. Asymmetric Dice Loss (Binary Head): Down-weights false positives to bias optimization toward recall, ensuring sparse defects are not missed.
2. One-Sided Online Self-Correction (Semantic Head):
   • Mechanism: If the student predicts a defect class with high confidence (above threshold $\tau$) while the teacher (SAM) labels the region as background, the supervision is selectively relaxed.
   • Logic: This allows the student to override unreliable background labels to recover missed defects (false negatives) without allowing the model to drift arbitrarily (only background labels are corrected, not defect labels).
   • Loss: Standard Cross-Entropy is applied to the corrected targets.

3. Key Contributions

1. Noise-Robust Distillation Pipeline: A framework that learns pixel-level segmentation from bounding boxes by explicitly modeling the unreliability of foundation model pseudo-labels.
2. Hierarchical Architecture: A novel student design combining frozen self-supervised transformer features (for stability) with a dedicated CNN detail branch (for sparse structure preservation).
3. Asymmetric Self-Correction: A training mechanism that enables the model to recover defects missed by the teacher or missing from box annotations, significantly improving recall without increasing false positives.
4. Efficiency: The use of BitFit adaptation reduces trainable parameters by 80% compared to fully trainable baselines while maintaining real-time inference speeds.

4. Experimental Results

Evaluated on a manually annotated wind turbine inspection benchmark (approx. 13,000 images) with only bounding box supervision during training.

• Performance Gains:
  • Anomaly mIoU: Improved by +6.97% over the strongest baseline (SegFormer-B2).
  • Binary IoU: Improved by +9.71%.
  • Recall: Online self-correction increased binary recall by +18.56%, directly addressing the critical issue of missed defects.
• Ablation Studies:
  • Removing the Binary Head caused the largest performance drop, highlighting the need for explicit structural localization.
  • Removing the Local Detail Branch reduced performance on thin structures.
  • Self-Correction showed minimal impact on validation (against noisy teacher) but massive gains on the test set (against ground truth), proving it recovers true defects the teacher missed.
• Efficiency:
  • Trainable Parameters: Reduced from ~20M (baselines) to 5.6M (Boxes2Pixels).
  • Inference: Achieved 161.4 FPS on an NVIDIA H100 with the lowest latency (6.20 ms).

5. Significance

• Industrial Applicability: The method solves the "annotation bottleneck" in industrial settings by leveraging cheap box labels and robust pseudo-labels, making high-quality defect segmentation scalable.
• Paradigm Shift: It moves away from treating foundation models as oracles. Instead, it treats them as noisy teachers and focuses on how the student learns from them (via architecture design and loss correction) rather than just how the labels are generated.
• Cost-Effectiveness: By freezing the backbone and using BitFit, the approach is highly efficient, requiring minimal compute and annotation budget for domain adaptation, which is critical for real-world deployment.

In conclusion, Boxes2Pixels demonstrates that robust defect segmentation is achievable from weak supervision by combining stable semantic representations, detail-preserving architectures, and intelligent noise-handling strategies, outperforming standard weakly supervised baselines significantly.