OilSAM2: Memory-Augmented SAM2 for Scalable SAR Oil Spill Detection

1. Problem Statement

The detection of oil spills using Synthetic Aperture Radar (SAR) imagery faces three critical challenges:

• Appearance Variability & Scale Heterogeneity: Oil spills exhibit diverse visual cues ranging from fine-grained texture/speckle to large-scale structural morphologies. Single-scale models struggle to capture these multi-level features simultaneously.
• Lack of Temporal Continuity: Unlike video tracking, real-world SAR oil spill monitoring often involves processing unordered collections of images rather than sequential video frames.
• Memory Drift in Foundation Models: Existing foundation models like SAM operate on single images and cannot reuse information across scenes. While SAM2 introduces a memory mechanism for temporal coherence, applying it directly to unordered SAR data causes semantic drift. The model may retrieve irrelevant information (e.g., specific sea states or speckle noise from a previous, unrelated image), leading to inaccurate segmentation.

2. Methodology: OilSAM2 Framework

The authors propose OilSAM2, a memory-augmented segmentation framework built upon SAM2 but specifically redesigned for unordered SAR data. The architecture consists of three core innovations:

A. Hierarchical Feature-Aware Multi-Scale Memory Bank

Instead of a flat memory structure, OilSAM2 constructs a structured memory bank that explicitly models three distinct feature levels derived from the image encoder:

1. Texture Level: Captures fine-grained details and speckle statistics essential for fragmented slicks.
2. Structure Level: Captures mid-level morphological patterns (e.g., elongated shapes).
3. Semantic Level: Captures high-level representations to suppress "look-alike" phenomena (e.g., low-wind areas).

• Mechanism: Each feature level interacts exclusively with its corresponding memory group via scale-wise attention, preserving level-specific representations and preventing information blurring.

B. Scale-Adaptive Memory Fusion

Since different oil spill morphologies and sea states rely on different feature levels, a static fusion strategy is insufficient.

• Process: Features retrieved from the three memory levels are mapped to a shared space.
• Adaptive Weighting: The model computes a level-wise response score (based on the $L_1$ norm of the retrieved features) to determine the contribution of each level for the current input.
• Fusion: These scores are normalized to generate dynamic fusion weights ($\gamma$), allowing the model to adaptively aggregate complementary information from texture, structure, and semantic levels before decoding.

C. Structure–Semantic Consistent Memory Update

To address the unordered nature of SAR data and prevent semantic drift, the authors replace the standard continuous memory update with a conditional update strategy.

• Reliability Cues: The system maintains lightweight prototypes for the current memory state. For a new input, it calculates:
  • Semantic Discrepancy ($\Delta_{sem}$): Measured via cosine distance between the current semantic descriptor and the memory prototype.
  • Structural Variation ($\Delta_{str}$): Measured via gradient magnitude differences in structural features.
• Update Logic: Memory is only refreshed if the discrepancies exceed specific thresholds ($\tau_{sem}, \tau_{str}$).
• Integration: When triggered, new memory entries are integrated via an Exponential Moving Average (EMA) to ensure smooth transitions and prevent abrupt shifts caused by noisy SAR conditions.

3. Key Contributions

1. Task-Adaptive Memory Design: The first application of memory-augmented SAM (SAM2) to unordered SAR oil spill monitoring, specifically addressing the lack of temporal continuity.
2. Hierarchical Memory Bank: A novel architecture that separates and manages texture, structure, and semantic features, enabling robust cross-image information reuse across varying scales.
3. Drift Mitigation Strategy: A structure–semantic consistent update mechanism that selectively refreshes memory based on reliability cues, effectively preventing the propagation of irrelevant scene-specific artifacts.
4. State-of-the-Art Performance: The framework achieves superior segmentation accuracy compared to CNNs, Transformers, and other SAM-based baselines.

4. Experimental Results

The model was evaluated on two public datasets: M4D (1,002 training images) and Deep-SAR Oil Spill (SOS) (8,070 patches).

• Performance on M4D:

  • OilSAM2 achieved a 72.62% mIoU, outperforming the previous best (OSDMamba at 70.25% and SAM-Oil at 69.52%).
  • It showed a 14.32% increase in Oil Spill IoU compared to SAM-Oil, demonstrating superior handling of complex spill morphologies.
  • It maintained competitive performance on other classes (ships, land, look-alikes), indicating strong generalization.

• Performance on SOS Dataset:

  • OilSAM2 achieved 84.20% mIoU on PALSAR data and 83.67% on Sentinel-1 data.
  • It outperformed classical CNNs (U-Net, D-LinkNet), advanced networks (DeepLabv3, CBD-Net), and the base Medical SAM2 model.
  • It demonstrated higher F1-scores and Recall, particularly in detecting small and fragmented slicks.

• Ablation Studies:

  • The full model (combining Multi-scale Memory, Scale-Adaptive Fusion, and Consistent Update) yielded the best results.
  • The Structure–Semantic Update was found to be crucial for stabilizing boundary delineation.
  • The Multi-scale Memory Bank was confirmed to be effective for capturing heterogeneous spatial patterns.

5. Significance

This paper bridges the gap between general-purpose foundation models (SAM2) and the specific constraints of remote sensing applications. By decoupling memory updates from strict temporal sequences and introducing a hierarchical, reliability-gated memory system, OilSAM2 enables scalable, robust, and accurate oil spill monitoring in real-world operational settings where data arrives as unordered collections. This approach offers a new paradigm for adapting video-based foundation models to static, unordered image analysis tasks in environmental monitoring.