🤖 AI

Remote SAMsing: From Segment Anything to Segment Everything

Remote SAMsing is an open-source, training-free pipeline that enhances SAM2 for large-scale remote sensing imagery by employing a multi-pass masking strategy and tile-boundary merging to resolve the quality-coverage trade-off and spatial fragmentation, thereby achieving near-complete object coverage (91–98%) and superior segmentation precision across diverse resolutions and modalities without modifying the underlying model.

Original authors: Osmar Luiz Ferreira de Carvalho, Osmar Abílio de Carvalho Júnior, Anesmar Olino de Albuquerque, Daniel Guerreiro e Silva

Published 2026-05-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Osmar Luiz Ferreira de Carvalho, Osmar Abílio de Carvalho Júnior, Anesmar Olino de Albuquerque, Daniel Guerreiro e Silva

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, high-resolution aerial photograph of a city or a farm. You want to use a super-smart AI (called SAM2) to draw outlines around every single object in the picture—every car, every house, every tree, and every field.

The problem is that this AI was trained to look at small, standard-sized photos. When you feed it a massive image, two big things go wrong:

The "Picky Eater" Problem: The AI is very strict. If it's not 100% sure an object is there, it ignores it. If you tell it to be less picky to catch more objects, it starts drawing messy, blurry outlines. It's a trade-off: either you get perfect outlines for a few things, or messy outlines for many things.
The "Puzzle Piece" Problem: Because the image is too big, you have to chop it into smaller squares (tiles) to process them. But when you cut a picture of a house in half, the AI sees two half-houses. When you try to put the pieces back together, the outlines don't match up, and the house looks like it's been shattered.

The paper introduces a new tool called Remote SAMsing. Think of it as a smart workflow that fixes both problems without needing to retrain the AI or change how it works.

How It Works: The "Paint-Over" Strategy

1. Solving the "Picky Eater" (Coverage)
Instead of trying to find everything in one go, Remote SAMsing plays a game of "hide and seek" in multiple rounds:

Round 1: The AI looks at the image with very strict rules. It finds the most obvious, clear objects (like a bright red car) and draws perfect outlines around them.
The Magic Trick: Once an object is found, the system paints that area black in the image. It's like covering the found items with black tape so the AI can't see them anymore.
Round 2: The AI looks at the remaining parts of the image. Because the obvious stuff is covered, the "hidden" or harder-to-see objects (like a small shed in the shadows) now stand out more clearly against the black background. The AI finds them and paints them black too.
Relaxing the Rules: If the AI gets stuck and can't find anything new, the system gently loosens its strict rules just a tiny bit to catch the remaining tricky bits.

The Result: By peeling away the easy objects first, the system captures almost everything (91–98% of the image) without sacrificing the quality of the outlines.

2. Solving the "Puzzle Piece" (Spatial Consistency)
When the image is chopped into tiles, the AI might draw a house on the left side of a tile and the other half of the house on the right side of the next tile.

Contextual Padding: The system gives each tile a little "buffer zone" of extra image around the edges. This helps the AI see the whole object even if it's near the cut line.
The Best Match: When putting the tiles back together, the system looks at the edges. If a piece of a roof on the left tile touches a piece of a roof on the right tile, it asks: "Which neighbor do you fit with best?" It only connects them if they are the best match, preventing it from accidentally gluing a house to a nearby road just because they touch.

Why It's a Big Deal

It's a "Zero-Shot" Solution: You don't need to teach the AI anything new. It works on buildings, cars, and even agricultural fields using false-color satellite images, even though the AI was originally trained on regular photos of everyday objects.
The "Zoom" Effect: The researchers found that making the tiles smaller acts like a zoom lens. For tiny objects like cars in a city, smaller tiles helped the AI see them much better (improving detection from 56% to 85%).
It Scales: They tested this on a massive image (1.94 billion pixels—imagine a map of a whole city). The system handled it without getting slower or making worse mistakes.

The Bottom Line

Remote SAMsing is like a smart assembly line for map-making. It takes a giant, messy image, breaks it down, cleans it up piece by piece, and stitches it back together so that every object is clearly defined. It allows scientists and planners to use powerful AI to map the world without needing to be experts in how the AI's brain works.

1. Problem Statement

The application of the Segment Anything Model 2 (SAM2) to large-scale Remote Sensing (RS) imagery faces two critical bottlenecks that prevent its use in production environments:

The Quality-Coverage Trade-off: SAM2's Automatic Mask Generator (AMG) filters predictions based on predicted Intersection over Union (IoU) and stability scores.
- Strict thresholds yield high-quality masks but leave large portions of the image unsegmented (low coverage, often 30–68%).
- Relaxed thresholds increase coverage but introduce low-quality, noisy masks.
- Existing single-pass approaches cannot simultaneously achieve high coverage and high precision without manual tuning.
Spatial Inconsistency (Tiling Artifacts): RS images often span tens of thousands of pixels, requiring them to be divided into tiles (e.g., 1024×1024) for processing.
- Objects crossing tile boundaries are fragmented.
- Standard stitching methods often fail to reconcile these fragments into a globally consistent segmentation, leading to "mega-segments" (spurious fusions) or disjointed objects.

2. Methodology: Remote SAMsing

The authors propose Remote SAMsing, an open-source, zero-shot pipeline that solves these problems without modifying SAM2's architecture or requiring training data. The pipeline consists of three core components:

A. Multi-Pass Adaptive Segmentation (Solving Coverage)

Instead of a single pass, the pipeline iterates over each tile using a progressive scene simplification strategy:

Black Masking: After each pass, pixels belonging to accepted masks are painted black. This removes already-segmented objects, simplifying the scene for the next iteration.
Dense Grid Prompting: Prompts are placed on a uniform $k \times k$ grid, but points falling on already-segmented (black) pixels are discarded. This ensures prompts focus on residual, unsegmented areas.
Adaptive Threshold Decay: The pipeline starts with strict quality thresholds ( $\tau_{start}$ ). Thresholds are only relaxed (decremented by $\Delta$ ) when coverage gains stagnate (fall below a threshold $\epsilon$ ). This ensures the most salient objects are captured first with high precision, while later passes capture residual areas with lower thresholds.

B. Spatial Consistency via Contextual Padding and Merge (Solving Fragmentation)

Contextual Padding: Tiles are extracted with an overlap ( $p$ pixels) on all sides. Segmentation is performed on the padded window, but only the central core is kept. This ensures objects near edges are fully segmented, allowing the merge algorithm to detect contacts across tile boundaries.
Parameter-Free Best-Match Merge:
- Instead of merging all touching segments (which causes spurious fusions), the algorithm uses a Union-Find structure.
- For every segment at a tile boundary, it identifies the neighbor with the highest contact area (best match).
- Segments merge only with their single best neighbor. This handles 1:1 splits and N:1 splits (one object fragmented on one side, whole on the other) without requiring tunable parameters.

C. Tile Size as an Implicit Scale Parameter

The authors identify that tile size ( $T$ ) acts as a scale parameter. Since SAM2 resizes inputs to 1024×1024, reducing the tile size effectively magnifies objects in the feature space, improving the detection of small objects (e.g., cars) without needing SAM2's built-in multi-scale mechanism (crop_n_layers).

3. Key Contributions

Pipeline-Level Solution: A novel approach that achieves near-complete coverage (91–98%) and high spatial consistency without fine-tuning the foundation model.
Iterative Simplification: The "black mask" strategy effectively breaks the coverage-quality trade-off by isolating difficult-to-segment regions for subsequent passes.
Robust Boundary Reconciliation: A parameter-free merge strategy that prevents the "mega-segment" problem common in tiled segmentation.
Systematic Evaluation: The first comprehensive analysis of SAM2 on large RS images across seven scenes, three spatial resolutions (5 cm to 4.78 m), and two spectral compositions (RGB and MNF false-color).

4. Experimental Results

The pipeline was evaluated on ISPRS Potsdam (5 cm GSD), Brasília (24 cm GSD), and Agri-BR (4.78 m GSD, MNF false-color).

Coverage: Increased from 30–68% (single-pass SAM2) to 91–98% across all scenes.
Detection Quality (Det@0.5):
- Discrete Objects: Excellent performance on buildings (95%), cars (82–93%), and agricultural fields (95–100%).
- Small Objects: Reducing tile size from 1,000 to 250 pixels improved car detection on Brasília from 56% to 85%, outperforming SAM2's native multi-scale mechanism.
- Amorphous Classes: Lower detection rates for "stuff" classes (e.g., low vegetation, impervious surfaces) due to diffuse boundaries, but high Average Segment Accuracy (ASA) (76–99%), indicating pixels are correctly assigned even if object boundaries are not perfectly reconstructed.
Comparison with Baselines:
- Outperformed SLIC and Felzenszwalb superpixel methods significantly in Boundary IoU (BIoU) and detection rates.
- SAM2-based segments align with object contours (learned structure) rather than just spectral gradients.
Scalability: Successfully processed a 1.94 billion pixel Potsdam mosaic (36,000 × 54,000 pixels) with 97% coverage and no quality degradation, processing in ~20 hours on a single GPU.
Generalization: Achieved 99.5% ASA on Agri-BR using MNF false-color imagery, despite SAM2 being trained only on natural RGB images.

5. Significance and Implications

Democratization of RS Analysis: Remote SAMsing removes the need for domain experts to manually tune complex SAM2 parameters (IoU thresholds, grid density). The pipeline is "plug-and-play" for large images.
Superpixel Generation for OBIA: The output serves as an ideal over-segmentation input for Object-Based Image Analysis (OBIA). The high BIoU ensures segments do not leak across class boundaries, making them suitable for downstream classification.
Zero-Shot Capability: The method proves that foundation models can be applied to diverse RS modalities (including false-color and varying resolutions) without retraining, provided the segmentation pipeline handles scale and coverage issues.
Future Work: The authors suggest that while the pipeline solves coverage and consistency, future improvements could focus on hierarchical multi-scale processing to better handle amorphous land cover and very high-resolution small objects.

In conclusion, Remote SAMsing transforms SAM2 from a patch-based research tool into a production-ready engine for large-scale remote sensing segmentation, solving the inherent trade-offs of coverage, quality, and spatial consistency through a clever, parameter-free pipeline design.