DFM: Difference Feature Modeling with Text-Guided Gated Contrastive Loss for Remote Sensing Image Change Captioning

Technical Summary: Difference Feature Modeling (DFM) for Remote Sensing Image Change Captioning

Problem Statement
Remote Sensing Image Change Captioning (RSICC) aims to automatically generate natural language descriptions of surface changes between multi-temporal remote sensing images. While existing methods typically employ an encoder-decoder architecture with autoregressive generation, the authors identify a critical limitation: model laziness. Current approaches tend to prioritize learning easily generated, frequent vocabulary patterns over capturing the discriminative visual differences between images. Furthermore, the standard autoregressive loss uniformly supervises all tokens, ignoring the varying difficulty of generating position-dependent content, which leads to captions that lack semantic precision regarding specific changes (e.g., object additions or disappearances).

Methodology: Difference Feature Modeling (DFM)
To address these issues, the paper proposes a novel framework, Difference Feature Modeling (DFM), which reframes the training paradigm by integrating cross-modal contrastive learning and change detection knowledge. The framework consists of three core components:

1. Text-guided Gated Contrastive Loss (TGCL):

   • Mechanism: A text encoder (pre-trained All-MPNet-base-v2) aligns change-descriptive captions with difference features extracted from the vision encoder.
   • Gating Mechanism: To prevent optimization conflicts caused by "no-change" samples (e.g., captions stating "the scene is the same"), a gating indicator ($\delta_i$) filters out pairs without actual changes from the contrastive loss calculation.
   • Objective: This loss explicitly constrains the vision encoder to focus on visually grounded changes relevant to the text, rather than generic image features. It operates bidirectionally (text-to-image and image-to-text) using a similarity matrix based on CLS token differences and sentence embeddings.

2. Joint Feature Modeling (JFM) Module:

   • Knowledge Transfer: The framework incorporates a pre-trained Change Detection (CD) model. The CD encoder's intermediate embeddings serve as stable priors to guide the vision encoder, enhancing change-relevant features while suppressing background noise.
   • Feature Fusion: The JFM module fuses multi-scale difference representations. It computes direct difference features ($\Delta v$) and spatiotemporal features ($v_{diff}$) from the Vision Transformer (ViT). These are injected with CD priors via Cross-Attention, refined through Bidirectional Self-Attention, and finally fused using a Weighted Fusion (WF) mechanism. This allows the model to capture comprehensive spatiotemporal variations at both pixel and structural levels.

3. Overall Optimization:
   The total loss function combines the TGCL ($L_{TGC}$) with the standard autoregressive generation loss ($L_{gen}$), balanced by hyperparameters $\alpha$ and $\beta$. This dual-objective approach ensures the model learns to generate fluent text while being strictly grounded in visual evidence of change.

Key Contributions

• Identification of Model Laziness: The authors highlight that existing RSICC methods over-rely on generic linguistic patterns due to single-loss autoregressive training. They propose TGCL and pre-trained CD knowledge transfer to enforce visually grounded change understanding.
• Novel Architecture: The introduction of the Joint Feature Modeling (JFM) module, which jointly captures pixel-level and structural difference features across multiple scales, enabling more accurate and context-aware captioning.
• Text-Guided Gating: The design of a gating mechanism within the contrastive loss to exclude non-change samples, ensuring that the model optimizes only on meaningful change instances.

Experimental Results
Extensive experiments were conducted on two benchmark datasets: LEVIR-CC and Dubai-CC.

• Performance: DFM achieved state-of-the-art results, outperforming previous methods (including CNN-based, Transformer-based, and Mamba-based architectures like RSCaMa).
  • On LEVIR-CC, DFM achieved a CIDEr-D score of 142.51 and an average score ($S^*_m$) of 81.40, surpassing the previous best (RSCaMa) by a significant margin.
  • On Dubai-CC, DFM achieved a CIDEr-D score of 84.65 and an $S^*_m$ of 49.21, showing particularly strong few-shot learning capabilities on the smaller dataset.
• Ablation Studies:
  • Removing the gating mechanism or using standard contrastive loss without masking degraded performance.
  • Using the CLS token difference ($\Delta_{cls}$) for alignment proved superior to raw difference features ($\Delta_v$).
  • The specific bidirectional self-attention with weighted fusion in the JFM module yielded better results than simple concatenation, gating, or bilinear fusion.
• Qualitative Analysis: Visual comparisons demonstrated that baseline models often generated generic phrases (e.g., "built along the road") derived from context, whereas DFM produced more precise descriptions of specific changes (e.g., "vegetation has been replaced by a residential area").

Significance and Claims
The paper claims that by moving beyond the single autoregressive generation paradigm, DFM successfully mitigates model laziness. The integration of text-guided contrastive learning and change detection priors forces the model to attend to critical visual differences. The authors assert that this approach leads to consistent improvements across multiple metrics, particularly in CIDEr-D, which measures the consensus of human-like descriptions. The work establishes a new direction for RSICC by emphasizing the alignment of visual difference features with semantic text representations, resulting in more interpretable and accurate change descriptions for applications in land planning and urban expansion monitoring.