Integrating Semi-Supervised and Active Learning for Semantic Segmentation

1. Problem Statement

Semantic segmentation, particularly in remote sensing, requires dense pixel-level annotations that are time-consuming, expensive, and require expert knowledge. While Semi-Supervised Learning (SSL) leverages unlabelled data and Active Learning (AL) selects informative samples for annotation, they are typically used independently.

• Limitations of existing SSL: Teacher-Student frameworks often suffer from over-reliance on the teacher model, while Cross Pseudo Supervision (CPS) can lead to model collapse as weights converge.
• Limitations of existing AL: Traditional AL often discards unselected unlabelled samples, failing to leverage their structural information. Furthermore, correcting pseudo-label errors usually requires manual re-annotation, which consumes the limited labeling budget.

The core challenge is to reduce annotation costs while maintaining high segmentation performance by effectively utilizing both labelled and unlabelled data without relying solely on manual intervention for error correction.

2. Methodology

The authors propose a Hybrid Active Semi-Supervised Learning Framework that integrates three core components:

A. Teacher-Student-Friend (TSF) Architecture

To address the limitations of standard Teacher-Student and CPS frameworks, the authors introduce a third model, the "Friend":

• Structure: Three models (Teacher, Student, Friend) share the same architecture (e.g., DeepLabv3+) but have different weights.
• Training Dynamics:
  • The Teacher weights are updated via Exponential Moving Average (EMA) of the Student.
  • The Student and Friend are trained using Cross Pseudo Supervision (CPS): The Student generates pseudo-labels for the Friend, and vice versa.
  • The Teacher does not supervise the Friend directly.
• Benefit: This decouples the Student from over-reliance on the Teacher and prevents the model similarity issues found in pure CPS, providing greater resilience during training.

B. Error Mask Decoder (EMD) for Uncertainty Detection

To identify regions where pseudo-labels are likely incorrect, the framework uses a dual-strategy approach:

1. Confidence-based: Uses the probability output of the Teacher model to find low-confidence regions.
2. EMD Module: A dedicated 3-layer CNN trained on labelled data to predict an Error Map. It learns the discrepancy between ground truth and pseudo-labels using multi-scale features (from the encoder and ASPP blocks) and the predicted probability map.

• Fusion: The final uncertainty map is generated by combining the EMD output and the confidence map using a logical OR operation, ensuring broad coverage of potential errors.

C. Pseudo-Label Auto-Refinement (PLAR)

Instead of manually labelling all uncertain regions, the system employs an automatic refinement mechanism based on the cluster assumption (pixels of the same class have similar features in high-density regions):

• Feature Extraction: Features are extracted from the Teacher model at two levels (ASPP output and decoder second-to-last layer).
• Distance Metrics: For pixels in erroneous regions, the system calculates Euclidean and Mahalanobis distances to the mean feature vectors of known classes within the same image.
• Voting Mechanism:
  • If both distance metrics agree on a class, the pseudo-label is automatically corrected.
  • If they disagree, the region is flagged for manual annotation.
• Budget Efficiency: This allows the model to correct "easy" errors automatically, reserving the manual labeling budget only for the most ambiguous and challenging cases.

3. Key Contributions

1. Novel Hybrid Framework: The integration of Active Learning with an enhanced SSL strategy (TSF) specifically tailored for remote sensing and natural imagery.
2. Teacher-Student-Friend (TSF): A robust architecture that combines the stability of EMA-based Teacher-Student models with the diversity of CPS, mitigating performance bottlenecks.
3. Pseudo-Label Auto-Refinement (PLAR): A cost-effective module that corrects noisy pseudo-labels using feature similarity within images, significantly reducing the need for manual re-annotation.
4. Dual-Strategy Uncertainty Detection: The combination of an Error Mask Decoder (EMD) and confidence-based uncertainty to comprehensively identify error-prone regions.

4. Experimental Results

The method was evaluated on CityScapes (natural images), ISPRS Vaihingen (remote sensing land cover), and RoadNet (road segmentation).

• SSL Performance (CityScapes): The TSF framework outperformed state-of-the-art SSL methods (MT, CPS, AEL, UniMatch) across various label ratios (1/16 to 1/2). Notably, it achieved a 2.31% mIoU gain over the AEL baseline at the 1/8 ratio.
• Hybrid AL Performance:
  • CityScapes: Using only 16% of labels (vs. 40% for baselines), the proposed method achieved 66.22% mIoU, outperforming all baselines including those using 40% labels.
  • Vaihingen: With 33% of labels, the method achieved 63.09% mIoU, surpassing the best baseline (DEAL at 40% labels) by a significant margin.
  • RoadNet: Achieved 88.75% mIoU with 33% labels, outperforming the best baseline (DEAL) by nearly 4 percentage points.
• Ablation Studies:
  • Adding PLAR to the TSF+AL framework provided a massive 5.82% mIoU boost on Vaihingen, proving the efficacy of auto-refinement.
  • Higher confidence thresholds for error detection generally improved performance by utilizing more of the labeling budget early in training to correct errors.

5. Significance

• Cost Reduction: The framework drastically reduces the human annotation burden by automating the correction of pseudo-labels for a large portion of uncertain data, reserving manual effort only for the most difficult cases.
• Domain Adaptability: It demonstrates strong generalization across both natural and remote sensing domains, addressing the specific challenges of remote sensing (spectral variability, large scales).
• State-of-the-Art Performance: The method consistently outperforms existing SSL and AL baselines, proving that integrating these paradigms with auto-refinement is a superior strategy for data-scarce semantic segmentation tasks.
• Practical Impact: The approach offers a viable solution for large-scale remote sensing applications where high-resolution imagery is abundant, but annotated data is scarce and expensive.