Task-Guided Multi-Annotation Triplet Learning for Remote Sensing Representations

1. Problem Statement

Remote sensing applications often require learning a shared representation that supports multiple downstream tasks simultaneously, such as semantic classification (identifying species) and spatial reasoning (estimating size, shape, and location via bounding boxes).

• The Challenge: These tasks rely on heterogeneous supervisory signals. Semantic labels impose categorical structure, while box-level attributes provide geometric constraints. These two types of information often conflict; semantic similarity does not always align with geometric similarity, and vice versa.
• Limitations of Existing Methods: Current multi-task triplet loss methods (e.g., Multi-Annotation Triplet Loss or MATL) typically combine losses from different annotation types using static, fixed weights (e.g., a scalar $\lambda$).
  • This approach requires extensive manual tuning.
  • It assumes a global balance that may not hold across different datasets or task difficulties.
  • It adjusts the magnitude of gradients rather than modifying which samples influence the representation, potentially allowing uninformative or conflicting sample relationships to distort the latent space.

2. Methodology

The proposed method, Task-Guided Multi-Annotation Triplet Learning (TG-MATL), replaces static loss weighting with a dynamic, task-aware triplet selection mechanism. Instead of adjusting loss magnitudes, it filters which sample triplets are used for training based on the mutual information between annotation types.

A. Architecture & Setup

• Backbone: The method utilizes frozen Vision Transformer (ViT) models (specifically DINOv2, CLIP, and MAE) as fixed feature extractors. This isolates the effect of the triplet selection strategy from encoder updates.
• Two-Stage Training:
  1. Stage 1: Triplet learning shapes the latent space using the frozen encoder.
  2. Stage 2: Linear task heads are trained on the resulting embeddings to evaluate downstream classification and regression performance.

B. Core Mechanism: Mutual Information (MI) Guided Selection

The core innovation is a sampling strategy that prioritizes triplets where semantic and geometric structures agree.

1. Relevance Scoring: For each sample, a relevance score ($m_i$) is calculated by summing the Mutual Information between each continuous geometric feature (e.g., box area, squareness) and the categorical class label.
   $$m_i = \sum_{f=1}^{F} [Y_{box}]_{if} \cdot MI(y_{box,f}, y_{class})$$
2. Triplet Selection Strategy: Within each class, samples are selected for triplet construction based on two criteria:
   • Top-Ranked ($T^{top}$): The top $p_{top}$ fraction of samples with the highest MI scores (most informative cross-task relationships).
   • Random ($T^{rand}$): A random subset of the remaining samples to maintain diversity and prevent overfitting to only the "easiest" relationships.
3. Loss Formulation: The Multi-Annotation Triplet Loss ($L_{MATL}$) is computed only on the triplets formed by these selected samples, rather than the entire dataset.
   $$L_{MATL-TG} = \sum_{T_{selected}} L_{MATL}(a, p, n)$$

3. Key Contributions

• Dynamic Task Balancing: The paper introduces a paradigm shift from loss weighting (adjusting scalar coefficients) to sample selection (filtering data based on task interaction). This removes the need for manual hyperparameter tuning of loss weights.
• Mutual Information as a Selector: It leverages MI to quantify the alignment between heterogeneous annotations (semantic vs. geometric), using this metric to identify the most "informative" sample relationships that should shape the shared representation.
• Isolation of Representation Effects: By freezing the foundation models, the study rigorously isolates the impact of triplet selection on the latent space, proving that better sample selection alone can improve downstream performance.

4. Experimental Results

The method was evaluated on the Animal Wildlife Image Repository (AWIR) dataset, which contains aerial images of wildlife with class labels and bounding box annotations.

• Comparison Baselines:

  1. Frozen embeddings (No Triplet Loss).
  2. Class-label Triplet Loss (CLTL).
  3. Hard-sampled CLTL.
  4. Standard MATL (Fixed 0.5 weight).
  5. Proposed TG-MATL.

• Key Findings:

  • Classification: TG-MATL significantly outperformed standard MATL and matched or exceeded hard-sampled CLTL in many configurations. It achieved high accuracy (e.g., 96.80% with DINOv2) by aligning the representation with classification objectives through task-relevant sampling.
  • Regression (Box Prediction): TG-MATL achieved the highest $R^2$ scores for box feature regression across all models (e.g., 0.64 with DINOv2).
    • Crucially, TG-MATL improved regression performance compared to standard MATL and CLTL, whereas hard-sampled CLTL often degraded regression performance due to its bias toward semantic separation.
  • Ablation on Sampling: Experiments varying the "Top Percent" and "Random Percent" showed that:
    • Higher random sampling improves classification (increasing intra-class diversity).
    • Lower random sampling (stricter selection of high-MI samples) is critical for maintaining geometric consistency in regression tasks.

5. Significance and Conclusion

• Overcoming Static Limitations: The paper demonstrates that static weighting schemes are suboptimal for multi-task remote sensing because they cannot adapt to the varying degrees of alignment between semantic and geometric data.
• Efficient Representation Learning: By selecting triplets that maximize the agreement between tasks, the method constructs a latent space that naturally supports both semantic classification and geometric reasoning without requiring complex gradient balancing or retraining of large foundation models.
• Future Directions: The authors suggest extending this approach to continuous geometric attributes (avoiding discretization) and applying it to broader multi-modal settings where heterogeneous data sources need a unified representation.

In summary, TG-MATL offers a robust, tunable-free approach to multi-task learning in remote sensing, proving that curating the data relationships is more effective than merely scaling the loss functions.