Continual Distillation of Teachers from Different Domains

This paper introduces Continual Distillation, a paradigm in which a student model sequentially learns from a stream of heterogeneous teachers without access to their training data, and proposes Self External Data Distillation (SE2D) to effectively counteract the forgetting of unknown knowledge during the transfer of unknown knowledge using external unlabeled data.

The Gist

Imagine you are trying to become the world's greatest expert by learning from a series of famous mentors. However, there is a catch: you can only speak with one mentor at a time, and once a mentor leaves, they vanish forever. You cannot return to ask them questions, and you have no access to the original textbooks they used to learn their craft.

This is the core problem the paper addresses, which it terms Continual Distillation.

Here is a breakdown of their idea, the problems they found, and their solution, using simple analogies.

The Setup: The Problem of the "Vanishing Mentor"

In the old days of AI, if a student model wanted to learn, it could access all the data (the textbooks) of previous teachers. But today, AI models (so-called "Foundation Models") are so huge and expensive that we cannot keep them all. We must learn from them one after another as they are released, and then lose access to the old ones.

The student model must learn from a stream of teachers:

1. Teacher A teaches about animals.
2. Teacher B teaches about insects.
3. Teacher C teaches about plants.

The student must learn from A, then from B, then from C, without ever seeing A or B again.

The Two Major Challenges

1. The Problem of the "Blind Spot" (Transferring Invisible Knowledge)
The teachers know things the student has never seen. For example, Teacher A might be an expert in "marine animals," but the student has only seen pictures of "land animals."

• The Paper's Discovery: If the student practices on a random set of images that neither the student nor the teacher has seen before (let's call this "External Data"), something magical happens. When the teacher looks at these random images, they show uncertainty or confidence. By observing how the teacher reacts to these unknown images, the student can actually learn about the domain of "marine animals," even though the student never directly saw a marine animal.
• The Metaphor: Imagine a master chef (the teacher) tasting a foreign, unknown fruit. Even if the student has never seen this fruit, the chef's reaction (e.g., "This tastes like a mix of lemon and honey") teaches the student the flavor profile of that fruit. This is called Unseen Knowledge Transfer (UKT).

2. The Problem of "Amnesia" (Forgetting Invisible Knowledge)
Here comes the bad news. When the student proceeds to learn from Teacher B (insects), they begin to forget what Teacher A taught them about marine animals.

• The Paper's Discovery: Since the student never directly saw the marine animals, this knowledge is fragile. As soon as new information arrives, this old "ghost knowledge" disappears.
• The Metaphor: It is like learning a new language. If you learned French from a book but never spoke it, and then immediately started studying German, you might forget the French words you only learned "by reading." This is called Unseen Knowledge Forgetting (UKF).

The Solution: "Self-External-Distillation" (SE2D)

The authors realized that standard methods try to memorize the teacher's answers but fail to safely preserve the "ghost knowledge." They proposed a new trick called SE2D.

How it works:
Every time the student finishes learning from a teacher, they take a "snapshot" (a checkpoint) of their brain.

• Normally, when learning from the next teacher, the student would practice everything.
• The SE2D Twist: When the student practices on the "External Data" (the random images no one knew), they also practice on their own previous snapshot.
• The Metaphor: Imagine you are a student. Before starting your new German course, you take a moment to review your old French notes specifically while looking at a random, foreign fruit. You ask yourself: "Based on my old notes, how would I describe this fruit?" This forces your brain to keep the French knowledge alive while you are busy learning German.

By doing this, the student stabilizes the "ghost knowledge" of previous teachers without needing to see the original teachers again.

What They Found (The Results)

1. The right kind of "randomness" is crucial: The "External Data" (the random images) must be related to what the teachers know to some degree.
   • If the teachers know about animals and the random images are of other animals, the student learns a lot.
   • If the random images are of trucks (completely unrelated), the student gets confused and forgets even more.
2. The Trade-off: There is a balance. If you focus too much on the new teacher, you forget the old one. If you focus too much on the old one, you don't learn the new one. SE2D helps find the "Goldilocks" zone where the student retains old knowledge while learning the new.
3. It works: In various tests (such as recognizing different cat breeds or digits), their method helped the student retain more about the "vanished" teachers than other standard methods.

The Conclusion

The paper introduces a new method for how AI can learn from a stream of teachers who disappear after use. They found that using "random" data helps the student learn things they have never seen, but it also causes the student to forget these things quickly. Their solution, SE2D, is like a memory exercise that forces the student to review their past lessons on these random data, ensuring they do not lose the valuable insights from teachers they can no longer reach.

Important Note: The authors warn that this "Unseen Knowledge Transfer" is a double-edged sword. If the random data is poor or biased, the student might accidentally learn bad habits or biases from the teacher without ever noticing. They suggest this needs further investigation, but they do not claim to have solved this specific risk yet.

Technical Summary

Technical Summary: Continuous Distillation of Teachers from Diverse Domains

1. Problem Definition: Continuous Distillation (CD)

The work introduces Continuous Distillation (CD), a new paradigm designed to address the challenges arising from the rapid evolution and storage costs of Foundation Models (FMs). In contrast to traditional Continual Learning (CL), where a model learns from a sequence of datasets, CD focuses on a single student model learning sequentially from a stream of teacher models.

Key Constraints and Challenges:

• Sequential Access: The student model learns successively from teachers $T_1, T_2, \dots, T_N$. Once a teacher is processed, it is no longer available, and its original training data are inaccessible.
• Data Unavailability: The original training data of the teachers are typically undisclosed, proprietary, or too large to be stored.
• Heterogeneous Expertise: The teachers are trained on different domains (e.g., one excels at animals, another at insects), although they share a partially overlapping area (e.g., ImageNet).
• Fixed Distillation Data: The student model is trained on a fixed dataset $D_S$ that does not change over time.

The authors decompose the fixed distillation dataset $D_S$ into two categories:

1. Internal Data (ID): Data known to all teachers (the common area, $D_i$).
2. External Data (ED): Data known to no teacher ($D_e$).

Identified Core Phenomena:

• Transfer of Unseen Knowledge (UKT): The phenomenon where a student acquires knowledge about domains it never saw during training, solely because the teacher possesses this knowledge and the student model is exposed to external data (ED) during distillation.
• Forgetting of Unseen Knowledge (UKF): The phenomenon where knowledge transferred by previous teachers regarding unseen domains is lost when the student model learns from subsequent teachers. This differs from traditional catastrophic forgetting, as the "forgotten" knowledge was never part of the student's own training data but was acquired via distillation.

The central challenge of CD is to optimize the trade-off between UKT (acquiring new unseen knowledge) and UKF (retaining previously acquired unseen knowledge).

2. Methodology: Self-External-Data Distillation (SE2D)

To mitigate UKF while preserving the benefits of UKT, the authors propose Self-External-Data Distillation (SE2D).

Mechanism:
SE2D adapts the concept of Self-Distillation (common in CL) to the specific constraints of CD. At each step $t$, the student model $S_t$ is optimized using two loss terms:

1. Teacher Distillation: Standard knowledge distillation from the current teacher $T_t$ to the student $S_t$ on the entire distillation dataset $D_S$ (both ID and ED).
2. Self-Distillation: Distillation from the previous checkpoint of the student $S_{t-1}$ to the current student $S_t$, but exclusively on the external data ($D_e$).

Loss Function:
The total loss is defined as:
$$L_{SE2D} = L_{KD}(S_t, T_t; D_S) + L_{KD}(S_t, S_{t-1}; D_e)$$

Rationale:

• Restricting self-distillation to $D_e$ is crucial. Applying it to $D_i$ would merely reinforce knowledge that is already stable across all teachers.
• By focusing self-distillation on $D_e$, the method specifically preserves the "fragile" knowledge transferred by previous teachers regarding domains the student model has never seen.
• This approach stabilizes learning across heterogeneous teachers without requiring access to previous teachers or their training data.

3. Main Contributions

1. Introduction of a Paradigm: The work defines Continuous Distillation and shifts the focus from data-centric CL to model-centric CL, reflecting the reality of evolving Foundation Models where previous versions become inaccessible.
2. Discovery of UKT and UKF: The authors demonstrate that using external data enables Transfer of Unseen Knowledge, allowing students to learn domains missing from their training data. Conversely, they identify Forgetting of Unseen Knowledge, where this acquired knowledge is lost during sequential learning.
3. Proposed Solution (SE2D): They present SE2D, a method that preserves logits on external data to mitigate UKF.
4. Empirical Validation: Extensive experiments across multiple benchmarks (CIFAR20, Digits, DomainNet) confirm that SE2D reduces UKF compared to standard distillation baselines and improves cross-domain generalization.

4. Experimental Results

The authors evaluated SE2D against baselines, including KL Divergence, Logits Standardization (LS), Medium-Difficulty Sampling (MDS), Decoupled Knowledge Distillation (DKD), and Standard Self-Distillation.

Key Findings:

• Necessity of External Data: Training exclusively on internal data results in the student model performing well only in the common area. Including external data is essential for UKT and significantly boosts performance on unseen domains.
• Trade-offs: While ED enables UKT, it can exacerbate UKF if not managed. Standard distillation methods often suffer significant performance drops on previous unseen domains once new teachers are introduced.
• Performance of SE2D:
  • On CIFAR20 with related external data, SE2D improved average accuracy on unseen domains by over 9% compared to baselines for specific tasks (e.g., Domain 1).
  • SE2D consistently outperformed standard self-distillation on older domains, demonstrating better preservation of transferred knowledge.
• Sensitivity to Domain Gap: The effectiveness of ED and SE2D depends heavily on the semantic similarity between the external data and the teacher's domains.
  • Related ED: Using semantically similar data (e.g., CUB birds for CIFAR20) leads to significant gains.
  • Unrelated ED: Using highly dissimilar data (e.g., MNIST digits for CIFAR20) can degrade performance and sometimes result in lower accuracy than using only internal data.
  • Teacher Quality: SE2D relies on the teacher providing high-quality supervision on the external data. If the teacher performs poorly on the external domain (low quality), the benefits of SE2D diminish.

5. Significance and Claims

The work claims that Continuous Distillation is a critical paradigm for the era of Foundation Models, as it addresses the practical impossibility of storing or re-accessing massive, evolving models and their training data.

• Knowledge Control: The work highlights that the provenance of distillation data is a primary lever for controlling which knowledge is transferred. The authors argue that the ability to transfer "unseen" knowledge (UKT) is a double-edged sword: it offers generalization opportunities but also introduces risks of anchoring unknown biases or uncontrolled knowledge in the student.
• Modest Limitations: The authors acknowledge that SE2D is not a universal solution. Its success depends on the domain gap between external data and the teacher being manageable, and the teacher must be competent on the external data. They note that identifying data outside a teacher's domain is non-trivial when data is generated to imitate training sets.
• Future Directions: The work suggests that UKT carries both opportunities and risks, particularly regarding unintended biases. Future work is proposed to explore larger models (language and multimodal) as well as the safety implications of uncontrolled knowledge transfer.

In summary, the work concludes that in a world of inaccessible, evolving teachers, the strategic use of external data and self-distillation on this data is essential to create robust student models that preserve knowledge across a sequence of heterogeneous teachers.