Technical Summary: Foundation-Assisted Active Learning for Object Detection Annotation

Problem Statement

Remote sensing object detection relies heavily on large-scale, high-quality annotated data, a process that is significantly more time-consuming and costly than in natural images due to dense small objects, large scale variations, and complex imaging conditions. While Active Learning (AL) aims to reduce annotation costs by prioritizing informative samples, existing methods face three critical challenges in remote sensing scenarios:

1. Coupled Uncertainty: Current methods often rely solely on detector prediction confidence, failing to explicitly distinguish between classification uncertainty and localization uncertainty. This is particularly problematic in the cold-start stage, where a detector's classification capability may improve quickly due to pretraining, but its localization remains noisy and unstable.
2. Pseudo-Diversity: High-recall candidate generation often leads to redundant or fragmented proposals. Diversity sampling based on these candidates can result in "pseudo-diversity," where the annotation budget is wasted on near-duplicate or over-fragmented instances rather than covering diverse object appearances.
3. High Interaction Costs: In the early stages of annotation, even with reliable category predictions, annotators must frequently correct large localization errors in detector-predicted bounding boxes. This manual refinement burden offsets the efficiency gains of active learning.

Methodology

The authors propose a Foundation-model-collaborative Active Learning and Semi-automatic Annotation Framework. The core innovation is a Dual-Source Mechanism that integrates a visual foundation model pipeline with a standard object detector.

1. Dual-Source Mechanism

• SA-Source (Reference Localization Source): An offline, one-time precomputation pipeline using UPN (Universal Proposal Network) to generate category-agnostic, high-recall proposal boxes, followed by SAM2 (Segment Anything Model 2) to refine these proposals into tight masks and bounding boxes. This source provides stable geometric priors independent of the detector's current state.
• OD-Source (Detector Prediction Source): The current-round detector outputs, providing category predictions, confidence scores, and learnable semantic information.

2. Key Components

• Foundation-Enhanced Dual-Source Uncertainty (FDSU):
  • Instead of relying on detector confidence alone, FDSU matches bounding boxes between the SA-source and OD-source using Hungarian matching.
  • It jointly models localization agreement (IoU between matched boxes) and classification confidence.
  • An object-level uncertainty score is calculated using the harmonic mean of these two factors, emphasizing the "short-board effect" (low agreement or low confidence increases uncertainty). This improves sample selection stability during the cold-start phase.
• Object-Centric Diversity Sampling (OCDS):
  • To address pseudo-diversity, the method constructs object-level representations using DINOv2 features guided by SAM2 masks.
  • It calculates a novelty score based on the distance of candidate objects from class prototypes in the DINOv2 feature space.
  • A Fragmentation Suppression Term is explicitly added to the diversity score to penalize images containing over-fragmented instances (small-area fragments, spatial overlaps, or semantic duplicates), ensuring the annotation budget is spent on truly diverse scenes.
• Dual-Source Box Switching (DSBS):
  • Designed for the semi-automatic annotation interface during the cold-start stage.
  • DSBS replaces noisy detector-predicted boxes with the more stable, refined boxes from the SA-source (while retaining the detector's category labels).
  • This "geometry from SA, category from OD" approach significantly reduces the manual effort required to correct bounding box coordinates, accelerating the annotation loop.

3. Active Learning Loop

The framework operates in rounds:

1. Offline Precomputation: SA-source (UPN+SAM2) is cached once for all images.
2. Inference & Uncertainty: The detector runs on unlabeled images; FDSU filters the pool based on joint uncertainty.
3. Diversity Sampling: OCDS selects the final batch of images, maximizing coverage while suppressing redundancy.
4. Semi-Automatic Annotation: DSBS provides pre-labeled boxes for annotators to confirm or correct.
5. Update: The detector is fine-tuned on the newly labeled data, and the loop repeats.

Key Contributions

1. Framework Integration: A novel framework that integrates vision foundation models (UPN, SAM2, DINOv2) into the active learning and semi-automatic annotation loop for remote sensing.
2. Dual-Source Uncertainty: A method to derive sample uncertainty scores by matching detector predictions with foundation-model-generated references, improving stability in the cold-start stage.
3. Redundancy Penalty: A diversity sampling strategy tailored to foundation models that explicitly suppresses candidate fragmentation and pseudo-diversity.
4. Box Switching Mechanism: A DSBS module that leverages foundation models to provide stable geometric priors, reducing manual interaction costs and improving early-stage detection performance.

Experimental Results

The method was evaluated on four remote sensing datasets: DIOR, HRSC2016, DOTAv2, and FAIR1M.

• Performance: The proposed method achieved superior or comparable results to state-of-the-art active learning methods (e.g., Entropy, CoreSet, PPAL, DivProto, AL4FM) across most annotation budgets.
• Cold-Start Efficiency: The method demonstrated notably stronger performance in the low-budget regime. For instance, on HRSC2016 with only 20/30 labeled samples, the DSBS-enabled approach achieved ~57%/61% mAP, significantly outperforming baselines without box switching (22%/45%).
• Ablation Studies:
  • FDSU consistently outperformed single-source uncertainty metrics (Entropy, DSUE) in the early stages.
  • OCDS synergized effectively with FDSU, outperforming other diversity strategies (CoreSet, DivProto, MDE) across all budget points.
  • Foundation Pipeline: Analysis showed that UPN+SAM2 significantly improved recall and localization tightness compared to UPN alone, while the fragmentation suppression strategy reduced redundancy by approximately 34–38%.

Significance

The paper claims that integrating visual foundation models into active learning pipelines offers a new perspective for improving data construction efficiency in remote sensing. By decoupling localization stability from the detector's learning curve and providing stable geometric priors, the framework addresses the specific bottlenecks of cold-start active learning and semi-automatic annotation. The results suggest that this collaborative approach allows for the construction of high-quality remote sensing datasets with lower human annotation costs and faster convergence to usable detection performance.