Low-Data Supervised Adaptation Outperforms Prompting for Cloud Segmentation Under Domain Shift

1. Problem Statement

The paper addresses the critical challenge of adapting Vision-Language Models (VLMs), specifically those pretrained on natural images (like CLIP), to remote sensing imagery (satellite data).

• The Core Assumption: The dominant AI deployment paradigm assumes that prompt engineering (using domain-specific language) can bridge the gap between pretrained representations and specialized tasks without requiring labeled data or weight tuning.
• The Reality: Satellite imagery presents a "dual domain shift":
  • Visual Shift: Overhead perspectives, multi-spectral sensors, and amorphous atmospheric phenomena (clouds, haze) differ fundamentally from the object-centric natural scenes used in VLM pretraining.
  • Linguistic Shift: Meteorological terminology (e.g., "optically thin cirrus," "cloud shadow") is absent from the image-caption pairs used to train models like CLIP.
• Research Question: Can prompt engineering alone compensate for this severe domain shift in cloud segmentation, or is supervised fine-tuning necessary even with minimal labeled data?

2. Methodology

The authors conducted a controlled empirical study using CLIPSeg (a promptable segmentation model) on the CloudSEN12+ dataset (Sentinel-2 satellite imagery with expert pixel-level labels for clear sky, thick cloud, thin cloud, and cloud shadow).

Experimental Setup

• Baseline: Zero-shot CLIPSeg using simple class labels (e.g., "cloud").
• Prompt Engineering: Evaluated 60 distinct prompt variants across four strategies:
  1. Simple labels.
  2. Domain-specific terminology.
  3. Appearance descriptors.
  4. Contextual cues (including negative prompts like "not cloud").
• Supervised Adaptation: Compared two fine-tuning strategies across data budgets ranging from 0.1% (~8 images) to 100%:
  • Full Fine-Tuning (FFT): Updates all model parameters.
  • Low-Rank Adaptation (LoRA): Freezes the backbone and injects trainable low-rank matrices into the decoder's attention layers.
• Training Configuration: Used a composite loss function (Focal + Tversky + Boundary loss) to handle class imbalance and boundary definition. Hyperparameters were optimized via grid search.

3. Key Contributions

1. Systematic Evidence of Prompt Failure: The paper provides the first systematic proof that linguistic refinement cannot bridge the visual-linguistic gap in satellite segmentation.
2. Identification of the "Supervision Crossover": It identifies that supervised fine-tuning outperforms prompting with as little as 0.1% labeled data (~8 images).
3. Discovery of the "Supervision Dip": The authors found a phenomenon where fine-tuning on very small datasets (0.5–1%) temporarily degrades performance on spectrally ambiguous classes (thin cloud, cloud shadow) before recovering, a nuance masked by aggregate metrics.
4. Task-Structure vs. Compute Trade-off: It reframes the choice between LoRA and FFT not as a compute trade-off, but as a decision based on spectral ambiguity.

4. Key Results

A. Prompt Engineering is Ineffective

• Performance: Every engineered prompt variant underperformed the simple zero-shot baseline (0.255 mIoU).
• Worst Case: Some prompts scored as low as 0.07 mIoU (a 73% relative degradation).
• Reasoning: The CLIP embedding space is fundamentally misaligned with satellite spectral imagery. Negative prompts (e.g., "not cloud") failed because the model was never trained to interpret "not" as a semantic operator in this context; the token "not" carries no learned visual meaning.

B. Low-Data Supervised Adaptation is Superior

• Crossover Point: Both LoRA and FFT surpassed the zero-shot baseline with just 0.1% of the training data (~8 images).
• Efficiency: Labeling 5–10% of the data (~425–850 images) recovered approximately 85% of the maximum achievable mIoU.
• Diminishing Returns: Beyond 30% data, additional labels yielded marginal gains (<3%).

C. FFT vs. LoRA: The Role of Spectral Ambiguity

• Overall Performance: Full Fine-Tuning (FFT) consistently outperformed LoRA by 0.03–0.09 mIoU.
• Class-Specific Differences:
  • Spectrally Distinct Classes (Clear Sky, Thick Cloud): LoRA and FFT performed nearly identically.
  • Spectrally Ambiguous Classes (Thin Cloud, Cloud Shadow): FFT significantly outperformed LoRA (e.g., +12% mIoU for thin cloud).
• Explanation: LoRA's low-rank constraints are sufficient for clear boundaries but fail to capture the fine-grained, multi-dimensional reshaping required to distinguish subtle spectral overlaps (e.g., thin clouds vs. haze, shadows vs. dark terrain).

D. The Supervision Dip

• At 0.5–1% labeled data, fine-tuning caused a temporary drop in performance for ambiguous classes below zero-shot baselines.
• Cause: With too few examples, the supervision signal is strong enough to disrupt the zero-shot embedding structure but too weak to properly reshape it toward the target distribution.
• Implication: Aggregate mIoU can hide this degradation; practitioners must monitor per-class metrics at low data regimes.

5. Significance and Conclusion

• Paradigm Shift: The paper challenges the industry assumption that "prompting is the cheap alternative to labeling." For specialized domains like remote sensing, labeled data is not an expensive alternative; it is the necessary path.
• Practical Guidance:
  • Do not rely on prompt engineering for satellite segmentation; it fails completely.
  • Collect a small, high-quality labeled dataset (5–10% of available data) to achieve near-optimal performance.
  • Use LoRA for well-defined classes to save compute, but switch to Full Fine-Tuning if the task involves spectrally ambiguous classes (thin clouds, shadows) where performance is critical.
• Broader Impact: These findings suggest that for any domain with a severe visual-linguistic shift (e.g., medical imaging, industrial inspection), the "zero-shot" promise of foundation models may be illusory without even minimal supervised adaptation.