A Survey on Generative Modeling with Limited Data, Few Shots, and Zero Shot

This survey provides a comprehensive review and unified perspective on generative modeling under data constraints (limited-data, few-shot, and zero-shot), introducing novel taxonomies to categorize tasks and methods while analyzing over 230 papers to offer a practical roadmap for overcoming challenges like overfitting and guiding future research directions.

The Gist

The "Culinary School" of AI: A Guide to Making Art with Almost No Ingredients

Imagine you are a world-famous chef. You have a massive kitchen (a supercomputer) and a library of millions of recipes (a huge dataset). You can cook a perfect steak or a complex soufflé with your eyes closed. This is how most modern AI image generators work today: they are trained on millions of photos so they know exactly what a "cat," a "sunset," or a "face" looks like.

But what happens if you are dropped into a tiny, empty kitchen with only three ingredients? Or maybe you have zero ingredients and only a description of what you want to cook?

This is the problem this paper tackles. It's a massive survey (a "map") of how to teach AI to create new, realistic images when it doesn't have enough data to learn from. The authors call this Generative Modeling under Data Constraint (GM-DC).

Here is the breakdown of their findings, explained with simple analogies.

---

1. The Three Levels of "Hunger"

The paper categorizes the problem into three levels of difficulty, like different levels of a cooking challenge:

• Limited Data (The "Small Pantry"): You have 50 to 5,000 photos. It's not a lot, but you have something.
  • Analogy: You have a few apples and some flour. You can still make a pie, but you have to be very careful not to burn it.
• Few-Shot (The "Taste Test"): You have only 1 to 50 photos.
  • Analogy: You are given one single photo of a specific dog and asked to draw 100 different pictures of that same dog in different poses. You have to memorize the dog's face perfectly from just one glance.
• Zero-Shot (The "Blindfolded Chef"): You have zero photos. You only have a text description.
  • Analogy: Someone tells you, "Draw a picture of a cat wearing a tuxedo on the moon," but you've never seen a cat, a tuxedo, or the moon. You have to rely entirely on your imagination and general knowledge.

2. The Big Problem: "Memorizing" vs. "Learning"

When you give a powerful AI only a few photos, it gets confused. Instead of learning the concept of a cat, it just memorizes the specific cat in the photo.

• The Result: If you ask it to generate a new cat, it just spits out the exact same photo you gave it, or a slightly blurry version of it. It hasn't learned the "rules" of being a cat; it's just a photocopy machine.
• The Paper's Goal: How do we stop the AI from being a photocopier and make it a true artist, even with limited ingredients?

3. The Toolkit: How Do We Solve This?

The authors organized hundreds of research papers into a "toolbox" of strategies. Here are the main ones, explained simply:

A. Transfer Learning (The "Master Chef's Mentorship")

Instead of teaching the AI from scratch, we take a model that is already an expert (trained on millions of photos) and give it a tiny "refresher course" on your specific topic.

• The Analogy: Imagine a master chef who knows how to cook everything. You ask them to learn how to cook a specific local dish using only 5 ingredients. You don't teach them how to hold a knife; you just show them the 5 ingredients and let them adapt their existing skills.
• The Catch: Sometimes the master chef tries to use techniques from their old kitchen that don't fit the new ingredients (e.g., trying to make a steak out of tofu). The paper discusses how to stop the AI from "forcing" old knowledge onto new problems.

B. Data Augmentation (The "Magic Mirror")

Since you don't have enough photos, you create fake variations of the ones you have. You flip them, rotate them, change the colors, or cut them up.

• The Analogy: You have one apple. To make it look like you have a hundred, you take a photo of the apple, rotate it 90 degrees, make it red, then make it green, then cut it in half. You tell the AI, "Look, here are 100 different apples!"
• The Risk: If you do this too much, the AI might think "rotated apples" are a new species and start generating only rotated apples.

C. Natural Language Guidance (The "Descriptive Critic")

This is the newest and hottest trend. We use the AI's ability to understand text to guide the image creation.

• The Analogy: You don't show the AI a photo of a "sad clown." Instead, you tell a very smart assistant (like CLIP, a language model), "Make it look like a sad clown." The assistant checks the AI's drawing and says, "No, that looks too happy. Make the eyes droop more." The AI adjusts based on the text, not the photos.

D. Frequency Components (The "High-Definition Filter")

AI often struggles with fine details (like the texture of fur or the wrinkles in skin) because it focuses on the "big picture" (low frequency) and ignores the "tiny details" (high frequency).

• The Analogy: Imagine looking at a painting from far away. You see the colors and shapes. But if you walk up close, you see the brushstrokes. Some of these methods force the AI to look at the "brushstrokes" (the high-frequency details) so the image doesn't look blurry.

4. The "Sankey Diagram": The Map of the Future

The paper includes a giant, colorful flow chart (a Sankey diagram). Think of this as a subway map for AI researchers.

• It shows which "stations" (tasks) are popular.
• It shows which "trains" (methods) are taking people there.
• The Surprise: Right now, almost everyone is taking the "Transfer Learning" train. But the map shows that some stations (like "Zero-Shot" or "Subject-Driven" generation) are just starting to get built.

5. Why Does This Matter?

You might ask, "Why bother? Can't we just wait for more data?"
The answer is no, because in many real-world fields, you can't get more data:

• Medicine: You can't take 1 million photos of a rare disease because only 50 people in the world have it.
• Satellite Imaging: You might need to detect a specific type of damage on a bridge, but you only have 10 photos of that damage.
• Art: An artist might want to generate images in their specific style, but they only have a few sketches.

6. The Future: What's Next?

The authors point out three big things we need to work on:

1. Better "Mentors": We need to use the newest, biggest AI models (Foundation Models) as our starting point, not the older ones.
2. Remote Adaptation: We need to teach the AI to learn from things that are very different from what it knows (e.g., teaching a face-generator to make flowers). Currently, it fails miserably at this.
3. Better Testing: How do we know the AI did a good job if we only have 5 photos to compare it against? We need new ways to judge the quality of these "small data" creations.

Summary

This paper is a massive guidebook for the "Culinary School" of AI. It tells us that while cooking with a full pantry is easy, cooking with a single ingredient is hard. But by using smart tricks—like borrowing skills from master chefs, using text descriptions as guides, and carefully selecting our ingredients—we can teach AI to create beautiful, diverse art even when the data is scarce.

Technical Summary

1. Problem Statement

Generative modeling typically relies on large-scale, diverse datasets (e.g., millions of image-text pairs for Diffusion Models or 70k faces for StyleGAN) to learn underlying data distributions. However, many real-world applications in medicine, satellite imaging, and artistic domains face Data Constraints (GM-DC) where training data is scarce, expensive, or private.
The paper addresses the challenge of training generative models under three specific constraints:

• Limited Data (LD): 50–5,000 samples.
• Few-Shot (FS): 1–50 samples.
• Zero-Shot (ZS): 0 samples (relying solely on pre-trained knowledge or text prompts).

Key challenges in this regime include overfitting (memorizing training data), mode collapse (lack of diversity), frequency bias (failure to generate high-frequency details), and incompatible knowledge transfer (transferring irrelevant features from source to target domains).

2. Methodology and Taxonomy

The authors conducted a comprehensive review of 233 papers and introduced two novel taxonomies to systematically categorize the field:

A. Task Taxonomy (8 Categories)

The paper defines eight distinct GM-DC tasks based on input conditions and domain relationships:

1. uGM-1 (Unconditional, Limited Data): Training from scratch on limited data.
2. uGM-2 (Unconditional, Cross-Domain Adaptation): Adapting a pre-trained source generator to a target domain with few samples.
3. uGM-3 (Unconditional, Zero-Shot Adaptation): Adapting a pre-trained generator to a target domain using only text prompts (no target samples).
4. cGM-1 (Conditional, Limited Data): Training conditional generators (e.g., class-conditional) with limited data.
5. cGM-2 (Conditional, In-Domain Unseen Classes): Adapting to unseen classes within the same domain using few shots.
6. cGM-3 (Conditional, Cross-Domain): Adapting a conditional generator to a new domain with few labeled samples.
7. IGM (Internal Patch Distribution): Generating diverse variations of a single image by learning its internal patch statistics (e.g., SinGAN).
8. SGM (Subject-Driven Generation): Generating diverse images of a specific subject (e.g., a specific backpack) using few shots and text prompts (e.g., DreamBooth).

B. Approach Taxonomy (7 Categories)

The paper categorizes solutions into seven methodological families:

1. Transfer Learning: Leveraging pre-trained models. Sub-categories include:
   • Regularizer-based Fine-tuning: Freezing parts of the network or using constraints (e.g., EWC, CDC) to prevent catastrophic forgetting.
   • Latent Space: Mapping source latent codes to target distributions (e.g., MineGAN).
   • Modulation: Adding trainable layers (e.g., LoRA, adapters) to frozen backbones.
   • Natural Language-Guided: Using CLIP or vision-language models to guide adaptation (e.g., NADA, Textual Inversion).
   • Adaptation-Aware: Dynamically identifying and pruning incompatible kernels (e.g., RICK).
   • Prompt Tuning: Learning visual tokens to guide generation (e.g., VPT).
2. Data Augmentation: Applying transformations (image-level, feature-level, or diffusion-based) to expand the data distribution (e.g., ADA, DiffAugment).
3. Network Architectures: Designing lightweight or dynamic networks to prevent overfitting (e.g., FastGAN, ProjectedGAN).
4. Multi-Task Objectives: Adding auxiliary losses (contrastive learning, masking, knowledge distillation) to improve generalization.
5. Exploiting Frequency Components: Explicitly modeling high-frequency details to counter spectral bias (e.g., FreGAN, WaveGAN).
6. Meta-Learning: Learning "how to learn" from seen classes to adapt to unseen classes (e.g., MatchingGAN, FAML).
7. Modeling Internal Patch Distribution: Learning self-similarity within a single image to generate variations (e.g., SinGAN, SinDDM).

3. Key Contributions

1. Comprehensive Scope: The first survey to cover all generative model types (GANs, Diffusion, VAEs) and all GM-DC tasks and approaches, analyzing over 230 papers.
2. Novel Taxonomies: Introduced structured frameworks for both Tasks (8 types) and Approaches (7 types), providing a unified vocabulary for the field.
3. Visual Analytics: Created a Sankey diagram visualizing the interactions between tasks, approaches, and specific methods, revealing trends and gaps.
4. Critical Analysis & Empirical Comparison:
   • Provided quantitative comparisons (FID, Intra-LPIPS, CLIP scores) across representative methods for each task.
   • Identified that Transfer Learning dominates the field (77% of works in 2024), specifically shifting toward Natural Language-Guided and Modulation methods.
   • Highlighted the Sample Selection Challenge: Demonstrated that performance varies drastically based on which few samples are chosen for training.
5. Identification of Research Gaps:
   • Remote Domain Adaptation: Current methods struggle when source and target domains are semantically distant (e.g., Human Faces $\to$ Flowers).
   • Evaluation: Existing metrics (FID, KID) lose statistical significance in zero-shot/few-shot regimes; a holistic evaluation framework is needed.
   • Data-Centricity: Lack of focus on sample curation and selection strategies.

4. Results and Findings

• Trends: Transfer learning has become the predominant solution, growing from 29% of works in 2021 to 77% in 2024. Within transfer learning, there is a shift from simple fine-tuning to Natural Language-Guided approaches (59% in 2024) and Modulation (LoRA-based).
• Model Types: GANs still dominate (68%), but Diffusion Models are rapidly rising (30%), while VAEs remain niche (2%).
• Task Distribution: Unconditional generation is the most studied (84% of works), while Zero-Shot learning is under-explored (3%) but growing.
• Performance:
  • For uGM-1, DANI (adaptive noise injection) achieved the best FID scores.
  • For uGM-2 (Cross-domain), RICK (pruning incompatible kernels) outperformed others by balancing fidelity and diversity.
  • For SGM (Subject-Driven), DreamBooth offers the highest fidelity but is computationally heavy, while tuning-free methods like MoMA and BLIP-Diffusion offer competitive efficiency.
• Challenges: The paper empirically demonstrates that incompatible knowledge transfer (e.g., transferring "glasses" from a face generator to a flower domain) severely degrades quality in remote domain adaptation.

5. Significance and Future Directions

This survey serves as a critical roadmap for researchers and practitioners aiming to deploy generative AI in data-scarce environments. It moves the field beyond incremental improvements on standard setups.

Future Directions identified:

1. Foundation Models: Leveraging massive pre-trained models (e.g., Stable Diffusion 3.5, DALL-E 3) as the base for GM-DC rather than just GANs.
2. Remote Domain Adaptation: Developing robust methods for transferring knowledge between semantically distant domains (e.g., medical to artistic).
3. Holistic Evaluation: Creating unified frameworks that combine objective metrics with human evaluation, specifically tailored for low-data regimes.
4. Data-Centric Strategies: Focusing on sample selection and curation, as the quality of the few available samples is a critical determinant of success.
5. Zero-Shot for Evolving Concepts: Enabling models to generate concepts that did not exist during pre-training (e.g., recent events).

The paper concludes that while significant progress has been made, the field must address the fundamental limitations of knowledge transfer in remote domains and the lack of robust evaluation metrics to fully realize the potential of generative modeling under data constraints.