RSRCC: A Remote Sensing Regional Change Comprehension Benchmark Constructed via Retrieval-Augmented Best-of-N Ranking

1. Problem Statement

Traditional Change Detection (CD) in remote sensing focuses primarily on identifying where changes occur (pixel-level or object-level masks) but fails to explain what changed in natural language. Existing Change Captioning datasets typically provide global, scene-level descriptions of differences between image pairs.

This approach has two major limitations:

1. Lack of Fine-Grained Reasoning: Models are not trained to reason about specific, localized semantic changes (e.g., "Was a new house built in the bottom-left corner?").
2. Annotation Bottleneck: High-quality, fine-grained semantic annotation is costly, slow, and difficult to scale, limiting the availability of large-scale datasets for training multimodal models.

The paper addresses the need for a benchmark that shifts from global summarization to localized, change-specific question answering (QA), requiring models to perform region-grounded semantic reasoning.

2. Methodology

The authors propose RSRCC, a benchmark constructed via a hierarchical semi-supervised curation pipeline. The process transforms raw satellite image pairs into high-quality, localized QA pairs without requiring dense manual annotation for every sample.

A. Data Source and Candidate Extraction

• Base Data: The pipeline utilizes the LEVIR-CD dataset (high-resolution building change detection).
• Candidate Localization:
  1. Semantic Segmentation: A transformer-based model (ViT-L encoder) generates semantic masks for "Before" and "After" images.
  2. Connected Component Analysis: Differences between masks are analyzed to extract coherent object-level change regions ($\Delta x$).
  3. Filtering: Regions are filtered based on size, Intersection over Union (IoU), and the ratio of changed pixels to remove noise.

B. Hierarchical Validation Pipeline

To ensure semantic consistency and filter false positives (e.g., shadows, seasonal vegetation changes), the pipeline employs a three-stage validation process:

1. Fast Semantic Screening (Image-Text Encoder):

   • A remote-sensing-tuned SigLIP vision-language encoder processes cropped change regions.
   • It compares the visual patch against text prompts of semantic classes. If the observed change does not match the top-$k$ predicted classes with high confidence, the candidate is discarded.

2. Retrieval-Augmented Validation (LLM Judge):

   • For ambiguous cases where the encoder is uncertain, a Large Language Model (LLM) acts as a judge.
   • Retrieval: The system retrieves top-$k$ semantically similar, human-annotated examples from a gallery to serve as in-context references (RAG).
   • Scoring: The LLM scores the candidate region against these retrieved examples to determine if the change is semantically valid.

3. Best-of-𝑁 Ranking:

   • Instead of a single pass, the system generates multiple candidate interpretations or scores for ambiguous regions.
   • It applies a Best-of-𝑁 selection strategy: it samples $N$ candidates, scores them using the retrieval-augmented reward model, and retains only the highest-scoring candidate (or those above a confidence threshold).
   • This acts as an inference-time preference-guided selection, reducing reasoning bias by constraining the LLM to locally relevant, in-distribution examples.

C. Question-Answer Generation

Once a region is validated as a true semantic change:

• An LLM (Gemini-2.5-Flash) generates diverse Question-Answer pairs conditioned on the localized region and its validated semantic label.
• Formats:
  • Yes/No: Binary verification of a specific change.
  • Multiple Choice (MCQ): 4-option questions testing semantic discrimination.
  • Open-Ended: Descriptive questions for linguistic diversity.
• Crucially, the LLM does not determine if a change occurred; it only diversifies the language for changes already validated by the vision pipeline.

3. Key Contributions

1. RSRCC Benchmark: A new dataset containing 126,000 localized change QA pairs (87k training, 17.1k validation, 22k test). It is the first remote sensing benchmark explicitly designed for fine-grained, region-grounded reasoning rather than global scene description.
2. Hierarchical Semi-Supervised Pipeline: A scalable framework combining segmentation, vision-language screening, and Retrieval-Augmented Best-of-𝑁 ranking. This significantly reduces manual annotation costs while maintaining high semantic precision.
3. Theoretical Insight: The paper provides an analysis showing that group-restricted retrieval (retrieving examples from the same semantic cluster) reduces reasoning bias and preserves local decision boundaries, preventing the model from being misled by out-of-distribution examples.
4. High-Quality Data: The dataset covers diverse semantic classes (buildings, vegetation, roads, pools, etc.) with a long-tail distribution, suitable for training robust multimodal models.

4. Results and Evaluation

The authors evaluated the pipeline's quality and the benchmark's utility using several metrics:

• Human Agreement: The full pipeline (Segmentation + Encoder + Best-of-𝑁) achieved 98% human agreement, significantly outperforming baselines like "Encoder Only" (11%) or "Segmentation Only" (61%). This confirms the pipeline's ability to produce semantically consistent annotations.
• Model Performance:
  • Yes/No Questions: State-of-the-art models (Gemini-2.5-Pro) achieved 68.32% total accuracy.
  • Multiple Choice: Achieved 48.76% accuracy, indicating the task is challenging and requires genuine reasoning.
  • Open-Ended: High BERTScore (87.51%) indicates strong semantic alignment between generated and human-written captions.
• Robustness: The pipeline demonstrated high robustness to random crops (0.2% false positive rate on random noise) and different segmentation backbones (high IoU consistency between models).
• Comparison to LEVIR-CC: Qualitative analysis showed RSRCC successfully detects fine-grained changes (e.g., small roads, partial building modifications) that are often labeled as "no change" in existing datasets due to annotation incompleteness.

5. Significance

• Paradigm Shift: RSRCC moves the field from "where did it change?" (detection) to "what exactly changed and where?" (comprehension), bridging the gap between computer vision and natural language understanding in remote sensing.
• Scalability: The proposed semi-supervised pipeline offers a blueprint for constructing large-scale, high-quality multimodal datasets without prohibitive human labeling costs.
• Foundation for Future Research: By providing a benchmark for localized reasoning, RSRCC enables the development of models capable of detailed environmental monitoring, disaster response analysis, and urban planning, where understanding specific semantic changes is critical.
• Methodological Innovation: The integration of Retrieval-Augmented Generation (RAG) with Best-of-𝑁 ranking for data curation is a novel approach to resolving ambiguity in automated labeling, potentially applicable to other domains requiring high-fidelity semantic validation.

The dataset is publicly available at Hugging Face, fostering further research into fine-grained remote sensing change comprehension.