🤖 machine learning

Wisdom of Committee: Diverse Distillation from Large Foundation Models and Domain Experts

The paper proposes DiverseDistill, an interactive distillation framework that leverages a learnable question-answer mechanism to effectively align outputs from diverse foundation models and domain experts into a compact student model, achieving superior performance recovery without requiring teacher co-optimization or incurring inference overhead.

Original authors: Zichang Liu, Qingyun Liu, Yuening Li, Liang Liu, Anshumali Shrivastava, Shuchao Bi, Lichan Hong, Ed H. Chi, Zhe Zhao

Published 2026-06-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zichang Liu, Qingyun Liu, Yuening Li, Liang Liu, Anshumali Shrivastava, Shuchao Bi, Lichan Hong, Ed H. Chi, Zhe Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant but tiny apprentice (a small, efficient computer model) how to do a specific job, like recommending movies or identifying cats in photos. You have access to two types of mentors:

The "Super-Genius" (Foundation Model): This mentor is incredibly smart, knows everything about the world, and has read every book on the internet. However, they are huge, slow, and speak a very complex language that the tiny apprentice struggles to understand.
The "Specialist" (Domain Expert): This mentor is smaller, faster, and speaks the apprentice's language perfectly. They are great at the specific job but don't know much about the wider world.

The Problem:
If you try to teach the apprentice directly from the Super-Genius, the gap is too big. The apprentice gets overwhelmed and learns very little. If you only use the Specialist, they learn the job well but miss out on the Super-Genius's broader wisdom.

If you try to put both mentors in the room and just ask them to "average out" their advice (a common method), it often backfires. The Super-Genius's complex, confusing advice clashes with the Specialist's simple advice, and the apprentice ends up performing worse than if they had only listened to the Specialist.

The Solution: "DiverseDistill" (The Interactive Translator)
The authors of this paper propose a new way to teach called DiverseDistill. Instead of just letting the mentors shout advice, they introduce a smart Translator (called the Distillation Module) that sits between the apprentice and the mentors.

Here is how it works, using a creative analogy:

1. The "Question and Answer" Game

Imagine the apprentice is in the middle of a quiz.

The Translator looks at the apprentice's current understanding and asks the Super-Genius a very specific question tailored to their style of thinking. It then asks the Specialist a different question tailored to their style.
The Mentors answer these questions. Because the questions are phrased in a way each mentor understands best, they give high-quality answers.
The Translator then takes those answers and "translates" them back into the apprentice's native language so the apprentice can actually learn from them.

2. The "Smart Filter" (Saving Energy)

The Super-Genius is expensive to talk to (it takes a lot of computing power). The Translator is smart enough to know: "For this specific question, the Specialist is the perfect expert; the Super-Genius isn't needed."
So, the Translator skips asking the Super-Genius for that specific question. This saves about 30% of the computing time during training without losing any quality.

3. The "Ghost" Translator

Once the apprentice has finished learning, the Translator and the Mentors are thrown away. The apprentice is left alone, but now they are incredibly smart. They are the same size and speed as they were before, but they have absorbed the best of both worlds. There is zero extra cost when the apprentice goes out to do the actual job.

What the Paper Found

The researchers tested this on two main tasks:

Movie Recommendations: They tried to teach a small movie-recommender using a massive language model (the Super-Genius) and a larger movie-recommender (the Specialist).
- Result: Standard methods failed or made things worse. DiverseDistill allowed the small model to learn 114% of the gap between a bad student and the best teacher. It actually learned more than the best single teacher could have taught it alone.
Image Recognition: They tried to teach a small image-recognizer using a massive vision model and a specialist.
- Result: Again, standard methods struggled. DiverseDistill recovered 73% to 90% of the performance gap, consistently beating all other methods.

The Bottom Line

The paper claims that you can't just dump a bunch of different experts into a room and expect them to teach a small student effectively. You need an interactive system that knows how to ask the right questions to the right expert and translate the answers.

By doing this, they managed to compress a massive 76-million-parameter model down to a tiny 2-million-parameter model (a 38x compression) while keeping almost all the intelligence, without needing to change the big models or train them together. The "Translator" is only used during the learning phase and disappears afterward, leaving a lean, powerful student ready for the real world.

Technical Summary: DiverseDistill

Problem Statement

Knowledge distillation from large foundation models to compact, specialized domain models faces significant hurdles due to substantial gaps in model capacity, architecture, and input modality. Direct distillation often fails to close the performance gap; for instance, experiments show that distilling from a 76M-parameter language model to a 2M-parameter recommender recovers less than 40% of the performance gap.

Furthermore, simply introducing domain-specific experts to form a "diverse teacher committee" (combining foundation models with domain models) does not automatically yield improvements. Standard multi-teacher distillation methods, which typically rely on averaging soft labels or logits, fail when teachers are heterogeneous. Naively combining a foundation model (e.g., an LLM) with a domain expert (e.g., an embedding-based recommender) can degrade student performance below that of single-teacher distillation. This occurs because standard fusion mechanisms cannot resolve incompatible output spaces (e.g., vocabulary logits vs. ranking scores) or align disparate internal representations without co-training or architectural modifications.

Methodology: DiverseDistill

The authors propose DiverseDistill, an interactive distillation framework designed to transfer knowledge from a diverse committee of strictly frozen teachers to a student model. The framework operates without updating teacher parameters, modifying teacher architectures, or co-training teachers with the student.

Core Components

The framework introduces a learnable Distillation Module consisting of two main components:

Question Augmenter ( $F_Q$ ):
- Takes the student's representation ( $x$ ) and generates teacher-specific questions ( $q_i$ ) for each teacher $T_i$ .
- Uses learnable teacher embeddings to condition the generation, ensuring the query is phrased in the "language" of the specific teacher's domain.
- Projects these questions into the native input dimension of each teacher.
- Simultaneously computes dynamic teacher importance weights ( $w_i$ ) per sample, allowing the system to skip low-relevance teachers during training to reduce computational cost.
Answer Augmenter ( $F_A$ ):
- Receives the hidden states ( $p_i$ ) from the final layer of each teacher after they process the injected questions.
- Projects these heterogeneous responses into the student's hidden representation space, producing aligned "answers" ( $a_i$ ) that serve as soft targets.

Training Process

The training objective minimizes a total loss function comprising three terms:

Task Loss ( $L_{task}$ ): Standard loss between the student's prediction and ground truth.
Distillation Loss ( $L_{distill}$ ): Measures the divergence between the student's hidden states and the aligned teacher answers, weighted by the importance scores ( $w_i$ ).
Task Regularizer ( $L_{reg}$ ): A critical mechanism to prevent the Distillation Module from converging to trivial solutions. It forces the generated questions and answers to be task-relevant by passing them through prediction heads and comparing them to ground-truth labels.

Gradient Routing: The student ( $S$ ) is updated via $L_{task}$ and $L_{distill}$ . The Distillation Module ( $F_Q, F_A$ ) is updated only via $L_{reg}$ . Teachers remain frozen throughout.

Inference

After training, the Distillation Module and all teachers are discarded. The deployed student model is identical in size and latency to a model trained without distillation, incurring zero additional inference overhead.

Key Contributions

Identification of the Diversity Liability: The paper demonstrates that while diverse teacher committees are theoretically beneficial, standard multi-teacher methods fail to leverage them when teachers differ significantly in architecture and modality, often harming performance.
Interactive Question-Answer Framework: DiverseDistill introduces a mechanism to selectively extract and align knowledge by generating teacher-conditioned queries and projecting responses into a shared student space.
Strictly Frozen Teacher Compatibility: The framework operates without co-training, gradient propagation into teachers, or architectural surgery, making it applicable to pre-deployed foundation models and API-only access scenarios.
Dynamic Efficiency: A learned importance mechanism filters low-relevance teachers per sample, reducing training forward passes by approximately 30% without quality loss.

Experimental Results

The framework was evaluated on recommendation (MovieLens) and vision (CIFAR-10/100, ImageNet) tasks with heterogeneous committees (e.g., LLM + MLP, CLIP + ViT).

Gap Recovery: DiverseDistill consistently outperformed all single- and multi-teacher baselines. It recovered 73–114% of the teacher-student performance gap.
Surpassing Teachers: In the MovieLens recommendation task (38× compression), the distilled student achieved a 114.2% Gap Recovery Rate (GRR), meaning it surpassed the performance of the best individual teacher.
Failure of Baselines: Standard multi-teacher methods (e.g., simple averaging) often resulted in negative GRR (e.g., -15.0% on CIFAR-10), confirming that naive diversity is detrimental without proper alignment.
Compression: The method achieved up to 38× model compression in recommendation tasks and 3.6× in vision tasks while maintaining or exceeding teacher performance.

Significance and Claims

The paper argues that teacher-specific, interactive knowledge extraction is essential for effective multi-source distillation, particularly when bridging foundation models and specialized applications. By treating teachers as a diverse committee and using a learnable interface to translate their expertise, DiverseDistill overcomes the capacity and modality gaps that render standard distillation ineffective.

The authors claim that this approach enables the deployment of highly compressed models that retain the generalization capabilities of massive foundation models and the efficiency of domain experts, all without the inference cost penalty typically associated with distillation. The work highlights that diversity, when properly managed, transforms from a liability into a significant asset for model compression and performance.