🤖 machine learning

Improving Diversity in Black-box Few-shot Knowledge Distillation

This paper proposes a novel training scheme for generative adversarial networks that adaptively selects high-confidence synthetic images under teacher supervision to enhance data diversity and achieve state-of-the-art performance in black-box few-shot knowledge distillation.

Original authors: Tri-Nhan Vo, Dang Nguyen, Kien Do, Sunil Gupta

Published 2026-04-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tri-Nhan Vo, Dang Nguyen, Kien Do, Sunil Gupta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a young apprentice (the Student) how to be a master chef. Usually, you would do this by letting the apprentice watch the master (the Teacher) cook thousands of meals using a massive library of recipes and ingredients.

However, in the real world, things are often different. You might only have five ingredients to work with (the Few-Shot problem), and the Master Chef is a mysterious figure who won't show you their secret recipes or how they chop vegetables; they only tell you the final name of the dish they made (the Black-Box problem).

This paper, titled "Improving Diversity in Black-box Few-shot Knowledge Distillation," introduces a clever new way to train that apprentice under these strict limitations. Here is the breakdown using simple analogies:

The Problem: The "Boring" Apprentice

When you only have a few ingredients (images) and a mysterious teacher, the apprentice tries to learn by just copying those few examples.

The Issue: If you only show an apprentice five pictures of cats, they might think all cats look exactly like those five. They won't learn what a cat looks like when it's sleeping, running, or black instead of orange.
Previous Attempts: Other methods tried to "fake" more pictures by smudging two real pictures together (like mixing red and blue paint to get purple). But this often resulted in blurry, weird-looking images that didn't teach the apprentice anything new.

The Solution: The "Confidence Coach"

The authors propose a new system called DivBFKD. Think of it as a three-step training camp:

1. The Magic Generator (The WGAN)

They use a special AI tool called a WGAN (a type of "Generative Adversarial Network"). Imagine this as a counterfeit artist trying to draw fake pictures, and a detective trying to spot the fakes.

Normally, the artist only has the five real ingredients to copy, so the fakes look boring and repetitive.
The goal is to get the artist to draw new, diverse pictures that still look like real food.

2. The "Confidence Coach" (The Teacher's Role)

Here is the paper's big innovation. Since the Master Chef (Teacher) won't show their secrets, they act as a Confidence Coach.

The Coach looks at the fake pictures the artist draws.
If the Coach says, "I am 99% sure this is a pizza," then that fake picture is High-Confidence.
If the Coach says, "I'm not sure, maybe it's a pizza or a pancake," that picture is Low-Confidence and gets thrown away.

3. The "Adaptive Rulebook" (Adaptive Thresholds)

The paper noticed that the Coach is sometimes biased. For example, the Coach might be super confident about "Pizzas" but unsure about "Salads." If we use one strict rule for everyone, we might get too many Pizza fakes and no Salad fakes.

The Fix: The authors created an Adaptive Rulebook. They adjust the "confidence bar" for each category individually.
- Example: To be a "High-Confidence Pizza," the Coach needs to be 95% sure. To be a "High-Confidence Salad," the Coach only needs to be 75% sure.
- This ensures the apprentice gets a diverse mix of fake images, not just the ones the Coach is naturally good at.

How It Works in Practice

Generation Phase: The artist draws fake images. The Coach checks them. If a fake image passes the "Adaptive Rulebook," it gets added to a special "High-Confidence" pile.
The Feedback Loop: The detective (who spots fakes) is now trained on both the real ingredients and the "High-Confidence" fakes. This forces the artist to get even better at drawing diverse, realistic images that the Coach loves.
Distillation Phase: Finally, the apprentice (Student) is trained on the original 5 real images plus all these new, diverse, high-quality fake images.

The Results

The paper tested this on seven different "kitchens" (datasets like MNIST, CIFAR, etc.).

The Outcome: The apprentice trained with this new method became much smarter than apprentices trained with old methods.
The Analogy: Instead of memorizing 5 specific cats, the apprentice learned the concept of a cat so well that they could recognize a cat they had never seen before.
The Score: They achieved the best results (State-of-the-Art) compared to all other methods that try to teach with limited data and a black-box teacher.

Summary

In short, this paper solves the problem of teaching a student with very little data and a secretive teacher by:

Generating new, fake training data.
Using the teacher to filter out the "bad" fakes and keep only the ones the teacher is confident about.
Adjusting the rules for what counts as "confident" so that every type of object gets a fair share of training.

This creates a rich, diverse learning environment that allows the student to learn effectively, even when the teacher is a "black box" and the data is scarce.

1. Problem Statement

The paper addresses the challenge of Black-box Few-shot Knowledge Distillation (KD).

Context: Standard KD transfers knowledge from a large "teacher" model to a smaller "student" model. However, traditional KD requires a large training dataset and full access to the teacher's internal parameters (white-box).
Constraints: In real-world scenarios (e.g., distilling proprietary models like Google's FaceNet or ChatGPT), the student developer often has:
1. Black-box access: Only the teacher's final prediction probabilities are available; internal gradients, logits, or parameters are hidden.
2. Few-shot data: Only a very small number of labeled images ( $N$ ) are available for training the student, insufficient to represent the true data manifold.
Limitations of Existing Solutions: Current few-shot KD methods often rely on white-box teachers or use simple data augmentation (like MixUp) which produces synthetic images that lack diversity or realistic semantics. Other methods using Variational Autoencoders (CVAE) suffer from blurring artifacts and fail to explicitly promote diversity.

2. Methodology: DivBFKD

The authors propose DivBFKD (Diverse Black-box Few-Shot Knowledge Distillation), a two-phase framework designed to generate diverse synthetic images that mimic the teacher's original training distribution without accessing the teacher's internals.

Phase 1: Generation (WGAN with High-Confidence Selection)

The core innovation lies in training a Wasserstein Generative Adversarial Network (WGAN) to generate synthetic images that are diverse and high-quality.

High-Confidence Images: Instead of using all generated images, the method filters synthetic images based on the teacher's confidence. An image $\tilde{x}$ is considered "high-confidence" if the teacher's maximum predicted probability $c_{\tilde{x}}$ exceeds a threshold $\tau$ .
Adaptive Thresholds ( $\tau$ ): A fixed threshold is ineffective because teachers exhibit class-specific biases (e.g., predicting "trousers" with higher confidence than "shirts"). The authors propose adaptive thresholds calculated per class using the $q$ -quantile (e.g., $q=0.1$ ) of confidence scores from the few-shot real images. This ensures the threshold is relative to the teacher's behavior for each specific class.
On-the-Fly Adversarial Training:
- The WGAN generator ( $G$ ) creates synthetic images.
- The discriminator ( $D$ ) is trained to distinguish between real images (from the few-shot set $D$ ) and high-confidence synthetic images (selected from the previous iteration).
- Feedback Loop: By treating high-confidence synthetic images as "real" for the discriminator, the generator is incentivized to produce more diverse images that the teacher classifies confidently. This creates a positive feedback loop expanding the diversity of the generated set.

Phase 2: Distillation

Dataset Construction: A distillation dataset ( $D_{KD}$ ) is constructed by combining the original few-shot real images ( $N$ ) and a budget of synthetic images ( $M$ ) generated by the trained WGAN.
Class Balancing: To prevent bias, synthetic images are generated with class balancing via rejection sampling.
Training: The student network is trained on $D_{KD}$ using standard Cross-Entropy loss against the teacher's predictions (soft labels).

3. Key Contributions

Novel Training Scheme for WGAN: Introduces a mechanism to inject high-confidence images into the adversarial learning loop on-the-fly. This effectively uses the teacher as a supervisor to guide the generator toward the teacher's unknown training distribution.
Adaptive Thresholds: Proposes a class-specific, dataset-agnostic method to define "high-confidence" using quantile-based thresholds, mitigating teacher bias and preventing the selection of abnormal images.
Diverse Black-box Few-shot KD (DivBFKD): A complete framework that addresses the lack of diversity in synthetic data, a critical bottleneck in few-shot KD.
State-of-the-Art Performance: Demonstrates significant improvements over existing few-shot KD methods across various datasets and architectures.

4. Experimental Results

The method was evaluated on seven datasets (MNIST, FMNIST, SVHN, CIFAR10, CIFAR100, Tiny-ImageNet, Imagenette) with various architectures (LeNet, AlexNet, ResNet, VGG).

Performance Gains:
- Simple Datasets: DivBFKD outperformed the second-best black-box method (FS-BBT) by ~2% on FMNIST and CIFAR10. It improved upon Standard-KD by 17% on CIFAR10.
- Complex Datasets: On CIFAR100, DivBFKD achieved 59.91% accuracy, surpassing FS-BBT (56.28%) by ~3.6%. On Imagenette (high-res), it reached 86.48%, significantly closing the gap to the "Student-Full" upper bound.
- Cross-Architecture: The method remained robust even when the teacher and student had different architectures (e.g., ResNet teacher to VGG student), improving accuracy by 14–22% over Standard-KD.
Diversity and Quality Metrics:
- Coverage: Synthetic images covered 42% of the teacher's training image neighborhoods, compared to only 18% for few-shot images and 29% for standard WGAN outputs.
- FID/IS: Achieved the best Fréchet Inception Distance (13.96) and Inception Score (4.42) among baselines, indicating high diversity and quality.
Ablation Studies: Confirmed that both Adaptive Thresholds (AT) and Class Balancing (CB) are essential components, with AT providing the most significant boost in diversity.
Comparison with Data-Free KD: DivBFKD significantly outperformed data-free (zero-shot) KD methods, proving that even a small amount of real data (2K images) combined with diverse synthetic data is superior to having no real data.

5. Significance

This paper makes a significant contribution to the field of model compression and privacy-preserving AI:

Practical Applicability: It solves a critical real-world problem where organizations cannot share large datasets or model weights due to privacy or IP concerns.
Diversity Focus: It shifts the focus from merely generating any synthetic data to generating diverse and teacher-aligned data, which is the key to effective distillation in data-scarce environments.
Efficiency: The proposed method adds negligible computational overhead (simple thresholding and on-the-fly selection) while delivering state-of-the-art results.
Generalizability: The approach is architecture-agnostic and works across simple and complex image classification tasks, making it a versatile tool for deploying lightweight models in constrained environments.