Grounding Synthetic Data Generation With Vision and Language Models

This paper proposes a vision-language grounded framework for interpretable synthetic data generation and evaluation in remote sensing, introducing the ARAS400k dataset which demonstrates that augmenting real data with synthetic images consistently outperforms real-data-only baselines in semantic segmentation and image captioning tasks.

The Gist

Imagine you are trying to teach a robot how to understand satellite photos of the Earth. You want the robot to be able to point out where the forests are, where the farms are, and where the cities are. The problem? Real satellite photos are expensive to get, hard to organize, and there just aren't enough of them to teach the robot everything it needs to know.

This paper is like a recipe for cooking up a massive, high-quality "fake" meal to feed the robot, but with a special twist: the fake meal is so well-made that the robot learns just as well as if it ate the real thing.

Here is the story of how they did it, broken down into simple steps:

1. The Problem: The "Empty Pantry"

In the world of AI, data is food. If you want a smart AI, you need a lot of it. But in remote sensing (looking at Earth from space), getting labeled data (photos where someone has already drawn lines around the trees and buildings) is like trying to find a needle in a haystack. It's slow, expensive, and often unbalanced (there are tons of photos of grass, but very few of swamps or mangroves).

2. The Solution: The "Three-Chef Kitchen"

The researchers built a smart kitchen with three chefs working together to create a new dataset called ARAS400k.

• Chef 1 (The Artist): This chef uses a "Generative Model" (like a super-advanced version of an AI art generator). It looks at the real satellite photos and learns the patterns. Then, it starts painting new fake photos that look exactly like real Earth, but they are brand new creations.
• Chef 2 (The Cartographer): This chef is a "Segmentation Model." Its job is to look at the photos (both real and the new fake ones) and draw precise maps. It says, "Okay, this pixel is a tree, this one is a road, this one is water." This gives the AI a clear "answer key" for every picture.
• Chef 3 (The Storyteller): This is the coolest part. Usually, AI just sees pictures. But here, they used "Vision-Language Models" (AI that can see and speak). This chef looks at the map drawn by Chef 2 and the picture from Chef 1, then writes a story about it.
  • Example: Instead of just seeing a picture, the AI writes: "This is a mostly agricultural landscape where crops cover more than half the area, with some city buildings and open grasslands mixed in."

3. The Secret Sauce: "Grounding"

Why is this special? Usually, when people make fake data, they just generate a pretty picture. But this team made sure the fake data was "grounded."

Think of it like a blindfolded test.

• If you show a robot a fake picture of a forest, it might guess "forest."
• But because they have the map (the segmentation) and the story (the caption) that matches the picture perfectly, the robot can cross-check.
• The caption says, "There is 57% crop here." The map shows 57% crop. The picture shows crops.
• Because all three agree, the robot knows this fake data is trustworthy. It's not just a pretty picture; it's a logical picture.

4. The Result: A Giant Library

They ended up with a massive library called ARAS400k:

• 100,000 real satellite photos.
• 300,000 high-quality fake photos.
• Every single photo has a map and a written description.

They tested this library by training new robots on it. Here is what they found:

• The "Fake Only" Test: Robots trained only on the fake data did surprisingly well, almost as good as those trained on real data.
• The "Mix" Test: The real winners were the robots trained on a mix of real and fake data. They became smarter than robots trained on real data alone!
• The "Rare Item" Bonus: The fake data was especially good at teaching the robot about rare things (like swamps or specific types of shrubs) that were missing from the real data. It's like the kitchen cooked up extra portions of the rare ingredients the robot was starving for.

5. Why This Matters

Imagine you are a student studying for a test.

• Old way: You have a small textbook with a few examples. You memorize them, but you might miss the tricky questions.
• New way (This paper): You get the small textbook, plus a tutor who generates 300 new practice questions that are perfectly explained and graded. You practice on all of them. You don't just memorize; you understand the patterns.

In short: This paper shows that we can use AI to create its own training data, as long as we use other AIs to verify that the data makes sense. This solves the problem of not having enough real-world data and helps AI understand our planet better, faster, and cheaper.

Where to find it:
The team put all this data and code online for anyone to use, so other scientists can build even smarter Earth-observing robots.

Technical Summary

Here is a detailed technical summary of the paper "Grounding Synthetic Data Generation With Vision and Language Models" by Ç a˘glar and Temizel.

1. Problem Statement

Deep learning models in remote sensing require vast amounts of diverse data, but acquiring and annotating real-world data is often costly, time-consuming, and limited by class imbalance (e.g., rare land cover types). While synthetic data generation offers a solution, current approaches suffer from three main limitations:

• Lack of Interpretability: Existing evaluation metrics rely on latent feature similarity, which is difficult to interpret and does not always correlate with downstream task performance.
• Semantic Misalignment: Synthetic data often fails to align semantically with real images, and evaluation procedures lack transparency.
• Data Scarcity and Imbalance: Traditional remote sensing datasets are small and often suffer from high caption redundancy and under-represented classes.

2. Methodology

The authors propose a three-stage, vision-language grounded framework to generate, evaluate, and utilize synthetic data for remote sensing tasks (specifically semantic segmentation and image captioning).

A. Data Acquisition and Pre-processing

• Source: ESA Sentinel-2 RGBNIR true-color images and WorldCover 2021 land cover maps.
• Alignment: Geo-aligned images and maps to extract 157,283 patches (256×256 pixels).
• Cleaning: Removed patches with >90% water (sun glare issues) and missing data.
• Class Merging: To address class imbalance, rare classes were merged (e.g., Snow $\to$ Barren, Mangrove/Wetland $\to$ Tree, Moss $\to$ Grass), resulting in 7 final classes: Tree, Shrub, Grass, Crop, Built-up, Barren, and Water.
• Final Real Dataset: 100,240 high-quality image-segmentation pairs.

B. The Generation Pipeline

The framework operates in three stages:

1. Training Foundation Models:
   • Generative Models: A custom StyleGAN3 architecture (updated for efficiency) combined with SPADE (Spatially-Adaptive Normalization) and a U-Net based discriminator. This allows for both unconditional image generation and conditional generation based on segmentation masks.
   • Segmentation Models: Trained on real data using architectures like UNet, DeepLabV3+, SegFormer, etc.
2. Synthetic Data Synthesis:
   • The trained generative models produce 300,000 synthetic images.
   • The trained segmentation models generate corresponding synthetic segmentation maps for these images.
3. Vision-Language Grounding & Captioning:
   • Composition Statistics: The segmentation maps are analyzed to calculate land cover percentages (e.g., "79% Grass, 15% Tree").
   • Caption Generation: Foundation models (Gemma3, Qwen3-VL) generate descriptive captions using three modalities:
     • Text-only: Based on composition statistics.
     • Vision-only: Based on image content.
     • Hybrid: Integrates visual content with composition statistics.
   • Evaluation: Captions are evaluated using CLIPScore (reference-free) and redundancy analysis to ensure semantic consistency and diversity.

3. Key Contributions

1. ARAS400k Dataset: A large-scale, open-source dataset containing:
   • 100,240 real and 300,000 synthetic images.
   • Paired semantic segmentation maps.
   • Over 2 million descriptive captions.
   • Significantly larger volume and lower redundancy (12.85%) compared to existing benchmarks like NWPU, RSICD, and UCMC.
2. Automated Context-Aware Framework: A pipeline that reduces manual labor by using segmentation-derived statistics to guide caption generation, ensuring captions are grounded in actual scene composition.
3. Vision-Language Integration: Demonstrates that integrating visual content with statistical composition data improves caption variety and reduces redundancy compared to text-only or vision-only approaches.
4. Proven Downstream Utility: Provides experimental evidence that synthetic data augmentation effectively addresses class imbalance and improves model performance.

4. Experimental Results

A. Dataset Quality

• Redundancy: ARAS400k achieved a caption redundancy of 12.85%, significantly lower than existing datasets (e.g., UCMC at 80.88%).
• CLIPScore: The dataset achieved competitive CLIPScores (~29.66), comparable to human-annotated benchmarks, validating the semantic quality of the automated captions.
• Visual Similarity: t-SNE and UMAP visualizations confirmed that synthetic samples cluster tightly with real data, indicating high visual fidelity.

B. Semantic Segmentation Performance

Models were trained on various data configurations (Real only, Synthetic only, Augmented).

• Synthetic-Only: Models trained exclusively on 300k synthetic images achieved competitive performance (e.g., Mean F1 ~74.79%) but were slightly inferior to real-data baselines (Mean F1 ~76.49%).
• Augmented Data (Real + Synthetic): This configuration yielded the best results, consistently outperforming real-data baselines.
  • Real + Unconditional (300k): Achieved the highest Mean F1 score of 77.66%.
  • Class Imbalance: The most significant performance gains were observed in under-represented classes (e.g., Shrub, Built-up), where synthetic augmentation alleviated data scarcity issues.

C. Captioning Performance

• Hybrid Models: The hybrid approach (Vision + Composition Statistics) yielded the highest caption variety and the lowest redundancy (0.47% for Gemma3-Hybrid on real data).
• CLIPScore: Vision-based models generally achieved higher CLIPScores, but the inclusion of composition statistics provided valuable structural grounding despite a slight dip in CLIPScore metrics.

5. Significance and Conclusion

• Scalability: The framework demonstrates a scalable method to expand remote sensing datasets by 4x without manual annotation, using automated pipelines.
• Interpretability: By grounding synthetic data in segmentation maps and language models, the authors provide a transparent evaluation method that correlates with downstream task utility.
• Practical Impact: The work proves that synthetic data is not just a substitute but a powerful augmentative tool. It is particularly effective for class-imbalanced scenarios, where adding synthetic samples for rare classes significantly boosts model accuracy.
• Future Directions: The authors suggest extending this framework to other domains (autonomous driving, medical imaging) and improving synthetic quality via super-resolution techniques.

The dataset and code are publicly available at Zenodo and GitHub, establishing a new benchmark for multi-modal remote sensing research.