🤖 machine learning

DiffuSAM: Diffusion Guided Zero-Shot Object Grounding for Remote Sensing Imagery

This paper proposes DiffuSAM, a hybrid pipeline that integrates diffusion-based localization cues with advanced segmentation models like RemoteSAM and SAM3 to achieve robust, zero-shot object grounding in remote sensing imagery, significantly outperforming existing state-of-the-art methods.

Original authors: Geet Sethi, Panav Shah, Ashutosh Gandhe, Soumitra Darshan Nayak

Published 2026-04-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Geet Sethi, Panav Shah, Ashutosh Gandhe, Soumitra Darshan Nayak

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking at a giant, high-resolution photo taken from space, showing a whole city or a forest. Someone asks you, "Can you point out the blue tennis court on the right side?" or "Find the big red ship in the middle."

Doing this is hard. The images are huge, the weather might be hazy, and there are thousands of tiny details. Usually, you'd need a super-smart computer trained specifically on thousands of examples of ships and tennis courts to do this. But what if you didn't have those examples? What if you wanted the computer to just "get it" immediately, like a human does?

That's exactly what the DiffuSAM paper is trying to solve. Here is the story of how they did it, explained simply.

The Problem: The "Blind" Detective

Traditional AI models are like detectives who have only ever studied one specific neighborhood. If you show them a picture of a different city, they get confused. They need massive amounts of training data (like a library of millions of photos) to learn what a "ship" or a "building" looks like. In the world of satellite imagery, getting those labeled photos is expensive and slow.

The Solution: A Three-Step Detective Team

The authors created a new system called DiffuSAM. Think of it not as one detective, but as a three-person detective team working together to find the object.

Step 1: The "Imaginative Artist" (The Diffusion Model)

First, the team brings in an AI Artist (a Diffusion model).

What it does: You give it the satellite photo and the text prompt ("Find the blue tennis court").
The Magic: Instead of just guessing coordinates, this artist imagines the scene. It takes the photo and digitally "paints" a red box around where it thinks the tennis court is.
The Catch: This artist is very creative but sometimes a bit wild. If the object is hard to see, the artist might "hallucinate" and paint a box around something that isn't there (like drawing a ship where there is only water). But, it's great at getting a rough idea of where to look.

Step 2: The "Photo Enhancer" (Pre-processing)

Before the team starts looking, they clean up the photo. Satellite images often look foggy or washed out because of clouds or pollution.

The Analogy: Imagine putting on a pair of high-tech sunglasses that cut through the fog and sharpen the edges of buildings. This step makes the image crisp so the other detectives can see clearly.

Step 3: The "Specialist Forensics Team" (The SAM Models)

Now that the Artist gave a rough red box, the team brings in two Specialist Forensics Experts (called SAM3 and RemoteSAM).

The Job: They zoom into the area the Artist pointed to. Their job is to trace the exact outline of the object, ignoring the background.
The Smart Switch: The system is smart enough to know which expert to use:
- If the area is huge (like a whole city block with many things), it calls RemoteSAM, an expert trained specifically on satellite photos.
- If the area is tiny (like a single small car), it calls SAM3, an expert great at fine details.
The Result: They take the Artist's rough red box and turn it into a perfect, tight green box that fits the object exactly.

Why is this a Big Deal?

Most AI systems need to be "taught" with thousands of examples before they can work. DiffuSAM is like a polymath—it can figure out new things immediately without needing a crash course.

No Training Needed: It works "Zero-Shot," meaning it can find objects it has never seen before, just by reading your text description.
Better Accuracy: The paper shows that by combining the Artist's creativity with the Forensics Team's precision, they found objects 14% better than the current best methods.

The Weakness (The "Hallucination" Risk)

The system isn't perfect. Because the first step relies on an "Artist" that uses imagination, sometimes it gets too creative. If you ask it to find a "unique ship" and there isn't one, it might invent a fake ship just to satisfy the request. The authors admit this is a flaw they hope to fix in the future.

The Bottom Line

DiffuSAM is a clever hack that combines the imagination of generative AI (which can "dream" up where things are) with the precision of specialized segmentation tools (which can "trace" exactly what is there). It's a way to make satellite computers smarter, faster, and able to understand our natural language requests without needing a massive library of pre-labeled photos.

1. Problem Statement

Remote sensing imagery analysis requires accurate object localization (grounding) for applications like urban planning and disaster response. However, developing specialized detection models is hindered by:

Data Scarcity: The high cost and difficulty of obtaining large-scale, annotated datasets for specific tasks.
Generalization Issues: Models trained on specific datasets often fail to generalize across different geographic regions or imaging conditions (e.g., haze, low contrast).
Zero-Shot Requirement: There is a critical need for data-efficient approaches that can perform accurate grounding using natural language prompts without extensive task-specific training.

While foundation models like SAM3 and RemoteSAM offer strong zero-shot capabilities, they may struggle with complex scene understanding or specific remote sensing nuances when used in isolation.

2. Methodology: The DiffuSAM Pipeline

The authors propose DiffuSAM, a hybrid pipeline that synergizes diffusion-based image editing with foundational segmentation models to achieve robust, zero-shot object grounding. The workflow consists of six key stages:

A. Pre-processing

To address atmospheric degradation (haze, low contrast) common in satellite imagery:

Color Space Conversion: Images are converted from RGB to CIELAB to separate luminance from chromatic components.
Enhancement: CLAHE (Contrast-Limited Adaptive Histogram Equalization) is applied to the luminance channel to enhance local contrast and reduce haze without amplifying noise.
Sharpening: The enhanced luminance is recombined with chromatic channels, followed by unsharp masking to improve edge definition.
Query Refinement: The input natural language query is processed by a small LLM (Qwen3-8b) to remove ambiguous instructions that might misguide the model.

B. Diffusion-Based Localization (Coarse Cue Generation)

Model: Qwen-Image-Edit (a diffusion-based image editing model).
Process: Given the pre-processed image and the refined text prompt, the model generates an edited image where the target object is highlighted with red bounding boxes.
Goal: This converts the text description into a coarse spatial cue, providing an initial estimate of the object's location without requiring annotated training data.

C. Initial Segmentation

The edited image (containing red boxes) is fed into SAM3.
SAM3 generates segmentation masks constrained to the highlighted regions, transforming the coarse diffusion cues into structured spatial representations.

D. Adaptive Refinement Strategy

To handle varying scene complexities, the pipeline dynamically selects the best segmentation model based on the size of the cropped region (relative to the original image):

Prompt Cleaning: Directional keywords (e.g., "left," "right") are removed from the prompt, as spatial relationships are now implicit in the cropped image.
Model Selection (Hyperparameter $p$ ):
- Large Regions (> $p\%$ ): Uses RemoteSAM. Large crops often contain multiple objects or complex backgrounds; RemoteSAM is trained specifically on satellite imagery grounding and handles these better.
- Small Regions ( $\le p\%$ ): Uses SAM3. Small crops usually imply precise localization, where SAM3's fine-grained segmentation capabilities are superior.
- Optimal Setting: Empirically, $p = 10\%$ yields the best performance.
Fallback Mechanism: If the selected model fails to produce a valid mask, the alternative model is invoked. If both fail, the system defaults to the diffusion model's initial bounding box.

E. Final Output

The final segmentation mask (from either RemoteSAM or SAM3) is used to compute the definitive bounding box, which serves as the pipeline's output.

3. Key Contributions

Hybrid Architecture: First integration of diffusion-based visual grounding (for coarse localization) with domain-specialized (RemoteSAM) and general-purpose (SAM3) segmentation models for remote sensing.
Zero-Shot Capability: The approach achieves high accuracy without task-specific training or large-scale annotated datasets, relying instead on the generative and foundational capabilities of pre-trained models.
Adaptive Strategy: A novel dynamic selection mechanism that chooses between RemoteSAM and SAM3 based on the scale of the region of interest, optimizing for both multi-object scenes and small-object precision.
Preprocessing Pipeline: A dedicated image enhancement step tailored for remote sensing conditions (haze/contrast) to improve model reliability.

4. Experimental Results

The method was evaluated on two benchmarks: VRS Bench (29k images, visual grounding) and NWPU-VHR-10 (800 images, 10 object categories).

Performance Metrics: Evaluated using Acc@0.5 (accuracy at IoU > 0.5), Acc@0.7, and mIoU.
Key Findings:
- NWPU-VHR-10: DiffuSAM outperformed the state-of-the-art RemoteSAM by a significant 14% in Acc@0.5 (0.8049 vs. 0.6624).
- VRS Bench: DiffuSAM achieved the highest Acc@0.5 (0.7913) compared to RemoteSAM (0.7812), SAM3 (0.2051), EarthMind, and Falcon.
- Overall: The hybrid approach consistently demonstrated superior or competitive performance across both datasets, validating the effectiveness of combining generative cues with segmentation refinement.

5. Significance and Future Work

Significance: DiffuSAM demonstrates that generative diffusion models can serve as effective "localization guides" for foundational segmentation models. This offers a scalable, data-efficient solution for remote sensing tasks where annotated data is scarce.
Limitations:
- Hallucinations: The diffusion model may generate synthetic objects if the target is absent or difficult to locate (Figure 3 in the paper).
- Axis-Aligned Constraints: Initial bounding boxes are axis-aligned, which may include unnecessary background or objects, potentially interfering with subsequent stages.
Future Directions:
- Controlling diffusion models to reduce hallucinations.
- Fine-tuning generative models specifically on remote sensing datasets to better handle atmospheric obstructions (haze) and improve the quality of the initial localization cues.

In conclusion, DiffuSAM represents a significant step forward in zero-shot remote sensing analysis, proving that a hybrid pipeline of generative and discriminative models can outperform specialized state-of-the-art models without the burden of extensive retraining.