VisText-Mosquito: A Unified Multimodal Dataset for Visual Detection, Segmentation, and Textual Explanation on Mosquito Breeding Sites

This paper introduces VisText-Mosquito, a unified multimodal dataset and framework that integrates visual detection, segmentation, and textual explanation to enable AI-driven proactive identification and analysis of mosquito breeding sites for disease prevention.

The Gist

Imagine a world where mosquitoes are like invisible thieves stealing our health, spreading diseases like malaria and dengue. For a long time, fighting them has been like trying to catch smoke with your bare hands: you know they're there, but finding their hiding spots (breeding sites) is hard, slow, and often requires armies of people walking around with nets and flashlights.

This paper introduces a new, super-smart digital detective team called VISTEXT-MOSQUITO. Think of it as giving public health workers a "superpower" that combines three things: Eyes (to see), A Brain (to understand), and A Voice (to explain).

Here is how it works, broken down into simple parts:

1. The "Eyes": Finding the Hidden Traps

Mosquitoes love to lay eggs in stagnant water. This water hides in weird places: inside old tires, broken flower pots, coconut shells, or even bottle caps.

• The Old Way: A human has to look at a photo and guess, "Is that water? Is that a tire?"
• The New Way: The researchers taught a computer (using a model called YOLOv9s) to act like a hawk. It scans thousands of photos and instantly spots these "danger zones." It's so good that it gets the answer right about 93% of the time. It doesn't just say "there's a tire"; it draws a box around it and says, "This is a danger zone."

2. The "Brain": Seeing the Water

Sometimes, a tire is just a tire. But a tire filled with rainwater is a mosquito nursery.

• The Challenge: It's hard for computers to tell the difference between a dry tire and a wet one, especially if the water is dark or hidden in shadows.
• The Solution: They used a special model (YOLOv11n-Seg) that doesn't just look at the object; it looks at the pixels. It's like a painter who can trace the exact shape of the water inside the tire, separating the wet parts from the dry parts. This helps them know exactly where the mosquitoes are likely to hatch.

3. The "Voice": Explaining Why

This is the most exciting part. Most AI tools just say "Danger!" but don't tell you why. If a doctor or a city planner sees a red box on a map, they need to know if it's a real threat or a false alarm.

• The Innovation: The team built a "talking AI" (based on a model called Mosquito-LLaMA3-8B).
• How it works: Instead of just pointing at a picture, this AI writes a short note.
  • Old AI: "Tire detected."
  • New AI: "This image shows an old tire filled with rainwater. Because tires hold water and are hard to empty, this is a perfect place for mosquito larvae to grow. It needs to be cleaned."
• The Result: The AI learned to speak like a human expert. It scored very high on tests that measure how well its writing matches human explanations. It's like having a junior health inspector who can look at a photo and write a report instantly.

Why is this a Big Deal?

Think of fighting mosquito diseases like fighting a fire.

• Before: You wait for the fire (the disease outbreak) to start, then you run in with hoses to put it out. This is expensive and dangerous.
• Now: This new system is like a smart smoke detector that doesn't just beep; it tells you, "There is a pile of dry leaves in the corner that could catch fire. Move them now."

By finding the breeding sites before the mosquitoes hatch, we can stop the disease before it starts. The paper emphasizes the theme: "Prevention is Better than Cure."

The "Recipe" for Success

The researchers didn't just guess; they cooked up a perfect recipe:

1. Gather Ingredients: They took 1,800+ photos of real breeding sites in Bangladesh, covering everything from sunny days to rainy nights.
2. Train the Chef: They taught the AI using these photos, correcting it when it made mistakes (like thinking a dry pot was wet).
3. Test the Dish: They let the AI look at new photos it had never seen before.
   • The "Eyes" (Detection) were incredibly accurate.
   • The "Voice" (Explanation) was so good that it beat other famous AI models that hadn't been trained specifically on mosquito problems.

The Bottom Line

This paper gives us a free, open-source toolkit (available on GitHub) that combines seeing, understanding, and explaining. It turns a complex, scary problem into something manageable. Instead of just reacting to disease outbreaks, communities can now use this AI to proactively clean up their neighborhoods, saving money and, more importantly, saving lives.

It's like giving every neighborhood a digital mosquito watchdog that never sleeps, never gets tired, and can explain exactly what it sees.

Technical Summary

1. Problem Statement

Mosquito-borne diseases (e.g., Malaria, Dengue, Zika) pose a severe global health risk, causing millions of illnesses and over a million deaths annually. Traditional control methods rely on manual inspection of breeding sites (stagnant water in containers), which is labor-intensive, slow, and often infeasible in large or inaccessible areas.

While computer vision has advanced, existing solutions face three critical limitations:

1. Lack of Specialized Data: Most datasets are unimodal (image-only) and lack the granularity to handle both object detection and water surface segmentation simultaneously.
2. Absence of Explainability: Current models detect hazards but cannot provide natural language explanations, limiting their utility in public health decision-making where interpretability is crucial.
3. Generalization Issues: Many models fail in complex, real-world environments (clutter, varying lighting) due to a lack of domain-specific training data.

2. Methodology

The authors propose VISTEXT-MOSQUITO, a unified multimodal framework that integrates visual data with textual reasoning. The methodology consists of three main pillars:

A. Dataset Construction (VISTEXT-MOSQUITO)

The dataset is the first of its kind to unify three tasks:

• Object Detection: 1,828 images with 3,752 bounding boxes across five container classes: coconut exocarp, vase, tire, drain inlet, and bottle.
• Water Surface Segmentation: 142 high-resolution images with 253 pixel-level masks focusing on "vase with water" and "tire with water."
• Textual Explanation: Each image is linked to a natural language triplet:
  • Question: Binary query (Is this a breeding site?).
  • Response: "Yes" or "No."
  • Explanation: A human-validated free-text justification (avg. ~36 tokens) describing visual cues (e.g., "stagnant water," "larvae").
• Preprocessing: Data was augmented (flipping, rotation, brightness adjustment) to expand the dataset to 4,425 images for detection and 331 for segmentation. Text was initially generated via GPT-4o and rigorously validated by humans.

B. Experimental Setup

• Hardware: Trained on an NVIDIA RTX 3090 Ti (24GB) with a 70/20/10 train/val/test split.
• Detection Models: Evaluated YOLOv5s, YOLOv8n, YOLOv9s, and RT-DETR ResNet-101.
• Segmentation Models: Evaluated YOLOv8x-Seg, YOLOv11n-Seg, and Mask R-CNN (ResNet-101 FPN).
• Vision-Language Models (VLMs):
  • Zero-shot & Few-shot: Tested LLaVA-1.5-7B, LLaMA3-8B-Vision, DeepSeek-VL2, InternVL-4B, Qwen2-VL, and closed-source models (Gemini-2.5-Flash, GPT-4o mini).
  • Supervised Fine-Tuning (SFT): Fine-tuned LLaMA3-8B-Vision on the dataset to create Mosquito-LLaMA3-8B.

C. Evaluation Metrics

• Visual: Precision, Recall, and mAP@50 (Intersection over Union threshold 0.5).
• Textual: BLEU, ROUGE-L, and BERTScore to measure fluency, relevance, and semantic similarity against human references.

3. Key Contributions

1. First Multimodal Benchmark: Introduced VISTEXT-MOSQUITO, the first dataset unifying object detection, water segmentation, and visual-linguistic explanation for mosquito breeding analysis.
2. High-Quality Annotations: Provided human-validated bounding boxes, segmentation masks, and explanatory text, enabling Explainable AI (XAI) in public health.
3. Comprehensive Benchmarking: Evaluated a wide spectrum of SOTA detectors, segmenters, and VLMs under zero-shot and few-shot settings, highlighting the limitations of unimodal approaches.
4. Domain-Adapted Model: Developed Mosquito-LLaMA3-8B, a specialized VLM that sets a new baseline for multimodal reasoning in vector surveillance.

4. Key Results

Object Detection

• Best Model: YOLOv9s achieved the highest performance with a Precision of 0.92926 and mAP@50 of 0.92891.
• Runner-up: YOLOv5s showed the best balance with the highest Recall (0.87595), making it suitable for minimizing false negatives in surveillance.

Water Surface Segmentation

• Best Model: YOLOv11n-Seg outperformed others with a Precision of 0.91587 and mAP@50 of 0.79795.
• Insight: One-stage segmentation (YOLO) proved more robust than two-stage (Mask R-CNN) in handling clutter and partial occlusions in real-world environments.

Textual Explanation Generation

• Fine-Tuning Superiority: The fine-tuned Mosquito-LLaMA3-8B significantly outperformed all zero-shot and few-shot baselines.
  • BLEU: 54.7 (vs. ~47.5 for best zero-shot).
  • BERTScore: 0.91 (vs. ~0.85).
  • ROUGE-L: 0.85.
• Few-Shot Impact: While few-shot prompting improved most open-source models, it caused performance degradation in Qwen2-VL, suggesting sensitivity to prompt construction. Closed-source models (Gemini) showed diminishing returns from few-shot due to strong inherent capabilities.
• Ablation Study: Confirmed that 3 epochs and full dataset usage were optimal; reducing data size or epochs led to underfitting or overfitting.

5. Significance and Future Outlook

• Public Health Impact: The framework operationalizes the "Prevention is Better than Cure" philosophy by enabling proactive, automated identification of breeding sites with human-readable justifications.
• Explainable AI: By generating textual explanations, the system bridges the gap between black-box AI detection and the trust required for field deployment by health officials.
• Scalability: The dataset and models provide a scalable foundation for vector control strategies, moving beyond simple detection to contextual understanding.
• Future Work: The authors plan to expand the dataset to diverse ecological regions, increase textual diversity, and explore prompt-adaptive models for localized intervention planning.

Availability: The dataset and code are publicly available on GitHub.