Open-Vocabulary vs Supervised Learning Methods for Post-Disaster Visual Scene Understanding

This paper presents a comparative evaluation of supervised learning and open-vocabulary vision models for post-disaster scene understanding across multiple datasets, concluding that while foundation models offer flexibility, supervised training remains the most reliable approach for accurately detecting small objects and delineating boundaries in cluttered disaster scenes when annotations are available.

The Gist

Imagine you are a first responder rushing into a city after a massive earthquake or a flood. The streets are a mess of debris, water, and smoke. You need to know instantly: Where are the survivors? Which buildings are safe? Where is the fire spreading?

To help you, you have a fleet of drones flying overhead, taking pictures. But looking at thousands of photos is impossible for humans. You need a computer to look at these images and tell you what's what.

This paper is a big "taste test" comparing two different types of computer brains to see which one is better at this life-saving job.

The Two Contenders

Think of the two methods being tested as two different types of students:

1. The "Specialized Intern" (Supervised Learning)

• How they learn: This student is given a massive stack of flashcards. Every card has a picture of a "flooded road" or a "collapsed building," and the teacher writes the exact name on the back. The student memorizes these specific cards until they can spot them instantly.
• The Catch: They are amazing at what they were taught, but if you show them something they've never seen (like a "burnt car" when they only studied "flooded cars"), they get confused. They need a lot of human teachers to make those flashcards first.

2. The "Polyglot Explorer" (Open-Vocabulary / Foundation Models)

• How they learn: This student didn't memorize flashcards. Instead, they read millions of books and watched millions of videos on the internet. They learned that "fire" looks like "orange and smoke" and "water" looks like "blue and wet." They understand the concept of things, not just specific labels.
• The Catch: They are very smart and can understand new words you give them on the spot. However, they haven't seen the specific chaos of a disaster zone before. They might know what a "car" is, but in a pile of rubble, they might mistake a piece of metal for a car.

The Experiment: The Disaster Olympics

The researchers put both students through a series of grueling tests using real drone photos from four different disasters:

• Floods: Trying to spot water vs. dry land.
• Earthquakes: Trying to spot damaged buildings vs. safe ones.
• Wildfires: Trying to spot smoke and flames.
• Search & Rescue: Trying to spot tiny people in a huge city.

They tested them on two main tasks:

1. The Coloring Book (Segmentation): Coloring every single pixel of the image (e.g., "This pixel is water, this one is a road").
2. The Scavenger Hunt (Object Detection): Drawing boxes around specific things (e.g., "Here is a person," "Here is a fire").

The Results: Who Won?

The Verdict: The "Specialized Intern" wins, but the "Polyglot" has potential.

Here is the breakdown in plain English:

• When the Intern wins: In almost every test, the student who memorized the specific disaster flashcards (Supervised Learning) was much more accurate. They were better at spotting tiny objects (like a person in a crowd) and drawing precise lines around messy, cluttered debris.

  • Analogy: If you need to find a specific needle in a haystack, the person who has studied needles for years will find it faster than the person who just knows what a "needle" looks like in a textbook.

• Where the Explorer struggles: The "Polyglot" (Open-Vocabulary) models were okay at finding big, obvious things (like a whole forest or a large lake). But when the scene got messy, or the object was small, they got lost. They often missed things or drew boxes in the wrong places because the disaster photos looked very different from the "normal" photos they learned on the internet.

  • The "Zero-Shot" Problem: When the Explorer tried to guess without any training on the specific disaster data, they performed terribly. It's like asking someone who only studied tropical beaches to navigate a blizzard.

• The "Cheat Code" (Transfer Learning): The researchers found a middle ground. If they took the "Polyglot" Explorer and gave them a little bit of training on the specific disaster photos (just a few hours of flashcards), their performance jumped up significantly. They didn't beat the Intern, but they got much closer.

The Big Takeaway

If you are building a system to save lives right now, you should use the "Specialized Intern."
If you have the time and resources to collect photos and label them for a specific disaster (like a specific city's flood), the supervised model is the most reliable, accurate, and safe choice. It handles the messy, confusing details of real-world disasters better.

The "Polyglot" models are exciting for the future because they are flexible. You don't need to retrain them for every new type of disaster. But right now, they are a bit too "dreamy" and not precise enough for the high-stakes, messy reality of a disaster zone.

In short: For the messy, chaotic reality of a disaster, a specialist who knows the specific terrain is still better than a generalist who knows the world. But with a little bit of training, the generalist can become a very strong helper.

Technical Summary

1. Problem Statement

Post-disaster situational awareness relies heavily on aerial imagery (UAVs and satellites) to assess damage from events like floods, earthquakes, and wildfires. However, automated interpretation faces significant challenges:

• Visual Complexity: Scenes contain extreme clutter, occlusions, scale variations, and ambiguous appearances (e.g., debris vs. shadows, smoke vs. clouds).
• Domain Shift: Visual characteristics vary drastically between different disaster types and geographic regions.
• Data Scarcity: High-quality, annotated datasets are limited, and existing supervised models rely on fixed label sets that cannot easily adapt to new or evolving disaster scenarios.
• The Gap: While Open-Vocabulary (OV) and Foundation Models (FMs) offer a promising alternative by leveraging vision-language pretraining to reduce dependence on fixed labels, their performance in the specific, high-stakes context of post-disaster aerial imagery remains under-evaluated compared to traditional supervised methods.

2. Methodology

The authors conducted a comprehensive comparative evaluation of Closed-Set Supervised Learning versus Open-Vocabulary (OV) approaches across two core tasks: Semantic Segmentation and Object Detection.

Experimental Framework

• Datasets: Four diverse benchmarks were used:
  • RescueNet: Post-earthquake damage segmentation.
  • FloodNet+: Flood scene understanding.
  • D-Fire: Wildfire and smoke detection.
  • LADD: Aerial pedestrian detection (Search and Rescue).
• Model Taxonomy:
  • Supervised Baselines:
    • Segmentation: DeepLabV3+, PSPNet, CCNet (CNNs); SegFormer, Mask2Former (Transformers).
    • Detection: YOLO series (v5, v8, v11, v26), RT-DETRv2.
  • Open-Vocabulary Models:
    • Segmentation: MaskCLIP, FC-CLIP, SegEarth-OV-3 (leveraging CLIP and SAM).
    • Detection: OWL-ViT, Grounding DINO, YOLOE (text-prompted variants).
• Evaluation Protocols:
  • Zero-Shot: Models were tested without fine-tuning on target datasets, using text prompts to define classes.
  • Transfer Learning: OV models were partially fine-tuned on target datasets to assess adaptability.
  • Metrics: Mean Intersection-over-Union (mIoU) for segmentation; mAP50 and mAP50:95 for detection.

3. Key Contributions

1. First Head-to-Head Benchmark: The study provides the first direct comparison of closed-set supervised pipelines against open-vocabulary alternatives under a unified protocol for both semantic segmentation and object detection in post-disaster scenarios.
2. Architectural Analysis: It analyzes the performance trends of CNNs vs. Transformers in cluttered, multi-scale aerial imagery.
3. Failure Mode Characterization: The paper identifies specific failure modes for OV models, particularly regarding small objects, occlusions, and fine-grained damage levels, offering practical insights for operational deployment.
4. Transfer Learning Insights: It quantifies the performance gap between zero-shot and fine-tuned OV models, demonstrating the necessity of domain adaptation.

4. Key Results

A. Semantic Segmentation

• Supervised Dominance: Supervised models significantly outperformed OV methods. On RescueNet and FloodNet+, top supervised models (e.g., CCNet, Mask2Former) achieved 76–79% mIoU, whereas zero-shot OV models struggled, achieving only 10–26% mIoU.
• Impact of Transfer Learning: Fine-tuning OV models improved performance drastically (e.g., MaskCLIP jumped from ~11% to ~29% mIoU on RescueNet), but they still lagged behind fully supervised models.
• Task Specifics: Supervised models excelled at fine boundary delineation and small object segmentation (e.g., vehicles in floodwaters). OV models struggled with ambiguous classes (e.g., distinguishing "minor" vs. "major" damage) and fine-grained distinctions.

B. Object Detection

• Small Object Challenge: Supervised detectors (especially YOLO variants and RT-DETR) achieved high accuracy (92–94% mAP50 on LADD for pedestrians). Zero-shot OV detectors performed poorly (6–61% mAP50), largely due to weak grounding of small targets in cluttered backgrounds.
• Domain Shift Sensitivity: Zero-shot OV models were highly sensitive to domain shifts. For example, OWL-ViT achieved only 6.2% mAP50 on LADD (pedestrians) in zero-shot mode.
• Transfer Learning Benefits: Fine-tuning OV models closed the gap significantly (e.g., Grounding DINO improved from 61% to 92% mAP50 on LADD), proving that OV models serve better as pre-trained initialization points than as drop-in replacements.

C. General Observations

• Clutter and Scale: Both paradigms struggled with heavy clutter and extreme scale variations, but supervised models handled them more robustly due to in-domain optimization.
• Class Imbalance: Supervised models were superior in handling rare classes and fine-grained taxonomies (e.g., specific damage levels), whereas OV models tended to predict only coarse, broad categories.

5. Significance and Conclusion

• Reliability of Supervision: The paper concludes that supervised training remains the most reliable approach for post-disaster aerial scene understanding when fixed label spaces and annotations are available. It is essential for high-precision tasks like small object detection and fine boundary delineation.
• Role of Open-Vocabulary: OV models are not yet ready to replace supervised systems in high-resolution disaster response. However, they offer a practical alternative when dense labels are scarce, provided they undergo transfer learning to adapt to the specific disaster domain.
• Future Directions: The authors suggest using foundation models for semi-automatic annotation to scale training data and improve cross-event generalization. They also highlight the need for integrating these systems with real-time AI event detectors and Human-Robot Interaction (HRI) frameworks for operational use by first responders.

In summary, while Open-Vocabulary models offer flexibility, supervised learning is currently indispensable for the robust, high-accuracy requirements of post-disaster visual analysis.