Technical Summary: Benchmarking Composed Image Retrieval for Applied Earth Observation

Problem Statement

The explosive growth of Earth Observation (EO) data has created massive satellite image archives that are difficult to navigate using traditional Remote Sensing Image Retrieval (RSIR) systems. Most existing RSIR approaches rely on single-modality queries (either an image or text), which limits expressivity when a user's intent involves targeted modifications beyond overall similarity. For instance, an analyst may need to find a location visually similar to a reference region but with a specific attribute change (e.g., a parking lot with lower occupancy) or a specific state change (e.g., the same site after a wildfire).

Remote Sensing Composed Image Retrieval (RSCIR) addresses this by combining a reference image with a natural language modifier to retrieve images that preserve relevant visual context while satisfying textual specifications. However, the transferability of modern composition methods to EO imagery, their behavior under unified evaluation protocols, and their effectiveness in operational workflows (particularly those requiring scene identity preservation) remain underexplored.

Methodology

The authors propose a unified benchmark and an application-oriented study to evaluate RSCIR in the EO domain.

1. Datasets and Evaluation Protocols

The study utilizes two distinct datasets reflecting different operational goals:

• PatternCom: Derived from PatternNet, this dataset focuses on attribute-modification retrieval. Relevance is defined by the "Same Class + Target Attribute" criterion (e.g., finding a swimming pool of a specific shape). It covers six attribute types: color, context, density, existence, quantity, and shape.
• xView2-CIR: A new dataset introduced by the authors, derived from xView2, focusing on change-centric retrieval for disaster monitoring. Relevance follows the "Same Scene/Location + Target State" criterion. Queries consist of a pre-event image and a modifier (e.g., "post-fire"), requiring the retrieval of the post-event image of the exact same geographic location. This isolates the challenge of preserving scene identity while applying a state change.

2. Vision-Language Backbones

The study evaluates six Vision-Language Models (VLMs), all utilizing a ViT-L/14 visual backbone:

• General-purpose: OpenAI CLIP, SigLIP, CLIP LAION-2B.
• Remote Sensing-adapted: RemoteCLIP, CLIP LAION-RS, SkyCLIP-50.

3. Composition Methods

The authors adapt and evaluate a range of composition strategies, ranging from simple score fusion to advanced inversion and diffusion techniques:

• Baselines: Unimodal (text-only, image-only) and simple multimodal fusion (averaging or multiplying scores).
• Training-free Methods:
  • WeiCom: Score fusion with Gaussian CDF calibration.
  • FreeDom: Memory-based textual inversion using a vocabulary of textual anchors.
  • BASIC: Training-free feature calibration involving centering, semantic projection, and multiplicative fusion.
• Textual Inversion:
  • Pic2Word: Learns an inverse of the text encoder to map image embeddings to pseudo-text tokens.
  • SEARLE: Performs test-time optimization of pseudo tokens with LLM regularization.
• Caption/LLM Pipelines:
  • CIReVL: Converts images to captions, then uses an LLM to merge with modifiers.
• Diffusion & Supervised:
  • CompoDiff: Casts composition as a diffusion process in the embedding space.
  • MagicLens: A supervised multimodal pooling method trained on web-mined triplets.

4. Adaptations for EO

To align these methods with remote sensing semantics, the authors:

• Created synthetic, domain-grounded vocabularies (RSText) using GPT-4o, ranging from 150 to 2,000 concepts, and a hybrid vocabulary (HybridText-23k) combining RSText with Open Images v7.
• Applied minimal domain-grounded modifications (e.g., prompt templates, vocabulary replacement) without extensive retraining on the test sets.

Key Contributions

1. Unified Benchmark: Established a standardized protocol for RSCIR on PatternCom, evaluating six VLM backbones and multiple composition methods with domain-grounded adaptations.
2. New Dataset (xView2-CIR): Introduced a change-centric benchmark for disaster and damage monitoring where relevance depends on both scene identity and target post-event state, addressing a gap in operational EO workflows.
3. Empirical Analysis: Provided practical guidance on when composed retrieval is useful, highlighting the distinct challenges between attribute-based and change-centric settings.

Results

Quantitative Findings

• Backbone Performance: Stronger general-purpose backbones (e.g., SigLIP) and remote-sensing-adapted models (e.g., RemoteCLIP) generally outperform generic CLIP variants. Domain adaptation yields gains, but improvements in general VLM architecture and pre-training data are equally critical.
• Method Performance on PatternCom (Attribute-based):
  • FreeDom emerged as the strongest performer across all backbones, achieving the highest average mAP (e.g., 46.61% on SigLIP). Its success is attributed to effective textual surrogates and domain-grounded vocabularies.
  • CIReVL and BASIC formed the next tier, performing particularly well on RS-adapted backbones.
  • WeiCom served as a competitive, lightweight baseline.
  • Methods relying on synthesized embeddings or pseudo-token inversion (e.g., Pic2Word, SEARLE, CompoDiff) were generally less competitive, often underperforming simple baselines or showing instability, suggesting sensitivity to domain shift in EO imagery.
• Method Performance on xView2-CIR (Change-centric):
  • WeiCom was the most robust method overall, achieving the best scores for both OpenAI CLIP and SigLIP.
  • FreeDom and BASIC were less effective here than on PatternCom. The study found that query-expansion mechanisms (visual proxies) in FreeDom introduced geographically different but visually similar scenes, which degraded performance when the "Same Scene" constraint was strict.
  • Simple score fusion (Text + Image) remained highly reliable.

Qualitative Insights

• Attribute Retrieval: FreeDom consistently respected both class identity and target attributes.
• Change Retrieval: A key failure mode identified was "scene drift," where methods retrieved images with the correct disaster state (e.g., post-fire) but from the wrong geographic location. This highlights that change-centric retrieval requires mechanisms that strictly anchor to the query location, which proxy-based expansion often fails to do.

Significance and Claims

The paper positions RSCIR not as a replacement for pixel-level change detection or damage assessment, but as a complementary tool for remote sensing workflows. Its primary value lies in:

• Controllable Archive Exploration: Enabling analysts to search large archives using semantically guided, targeted queries.
• Rapid Evidence Gathering: Facilitating the retrieval of visually similar historical cases or specific state changes without dense annotations.
• Workflow Integration: Serving as a lightweight interface for change-focused search, quality control (hard-negative discovery), and weak supervision.

The authors conclude that training-free composition methods (like FreeDom for attributes and WeiCom for change) provide strong, scalable baselines for EO retrieval. However, they emphasize that change-centric retrieval is a distinct setting from attribute-based editing; preserving scene identity fundamentally alters the method ranking and exposes weaknesses in strategies that rely heavily on visual similarity proxies. Future work should focus on expanding change-centric benchmarks, refining prompt design, and connecting scene-level retrieval with finer-grained localization.