KD4MT: A Survey of Knowledge Distillation for Machine Translation

1. Problem Statement

The field of Natural Language Processing (NLP) is trending toward ever-larger models (Large Language Models, LLMs), which increases computational costs and environmental impact. While Knowledge Distillation (KD) is traditionally viewed as a compression technique to reduce model size, its application in Machine Translation (MT) is more nuanced.

• The Misconception: A common assumption is that KD in MT is primarily for efficiency (compressing a large teacher into a small student).
• The Reality: The survey reveals that in nearly 50% of reviewed studies, teacher and student models have equal parameter counts. Thus, KD in MT often serves purposes beyond compression, such as knowledge transfer, regularization, data augmentation, and specialization for specific domains or languages.
• The Gap: There is a lack of unified evaluation practices, inconsistent reporting of datasets/metrics, and an over-reliance on English-centric translation directions. Furthermore, the risks of KD (e.g., hallucination, bias amplification, and reduced diversity) are often overlooked.

2. Methodology

The authors conducted a comprehensive survey of 105 research papers published up to October 1, 2025. The methodology involved:

• Categorization: The literature was synthesized based on two main axes:
  1. Methodological Contributions: How the distillation algorithm is engineered (e.g., filtering, expanding supervision, reframing).
  2. Practical Applications: Specific use cases in MT (e.g., Multilingual, Low-Resource, Domain Adaptation, Time-Sensitive).
• Quantitative Analysis: The authors analyzed parameter counts, compression ratios, dataset usage, and evaluation metrics across the surveyed papers to identify trends and gaps.
• Qualitative Analysis: They evaluated the effectiveness of different KD strategies (Word-Level, Sequence-Level, Feature-based) and their suitability for different architectural constraints (e.g., black-box teachers vs. white-box access).

3. Key Contributions & Technical Framework

A. Taxonomy of KD Methods in MT

The survey structures KD algorithms into three primary dimensions of engineering the supervision signal:

1. Selecting Supervision (Filtering):

   • Token-Level: Filtering uncertain tokens or using Top-1 information to focus on confident predictions (e.g., TIE-KD).
   • Sentence-Level: Refining the distillation dataset by filtering out poor teacher-generated translations or mixing them with gold references.
   • Representation-Level: Selecting specific layers or attention heads to transfer, though this is less common and harder to implement.

2. Expanding Supervision (Diversifying):

   • Multi-Teacher Distillation: Aggregating signals from multiple teachers or using structural diversity (e.g., Mutual Distillation) to enrich the student's learning.
   • Proxy/Auxiliary Tasks: Using intermediate tasks (like Masked Language Modeling) or Teacher Assistants to provide richer linguistic signals.
   • LLM Integration: Using LLMs to generate synthetic data, diagnose errors, or provide rationale-level guidance, moving beyond static teacher-student frameworks.

3. Reframing Supervision (Correcting Mismatches):

   • Addressing the Capacity Gap (student inability to fit teacher distribution) and Exposure Bias (training vs. inference mismatch).
   • RL-based Approaches: Using symmetric rewards (f-distill) or on-policy training (Generalized KD) to align the student's generation process with inference conditions, reducing exposure bias.

B. Application Areas

The survey details how KD solves specific MT challenges:

• Multilingual MT (MMT): Mitigating the "Curse of Multilinguality" by distilling strong bilingual teachers into a single multilingual student or scaling up massively multilingual models.
• Low-Resource MT (LRMT): Generating supervision signals where parallel data is missing using monolingual data, pivot languages, or synthetic data from LLMs.
• Domain Adaptation: Preventing catastrophic forgetting by separating general and domain-specific knowledge (Split-and-Distill approach).
• Time-Sensitive Applications:
  • Simultaneous MT (SiMT): Using "future-aware" KD to inject look-ahead context during training.
  • Non-Autoregressive MT (NAT): Crucially, KD is required here to provide deterministic, simplified targets that allow parallel decoding, acting as a regularizer to reduce target variability.

C. Practical Guidelines

The authors provide a decision framework for practitioners:

• Access Constraints: Seq-KD is the only option for black-box teachers (e.g., API-based LLMs) as it only requires output strings. Word-KD and Feature-based KD require access to internal probabilities or activations.
• Computational Budget: Word-KD is generally more efficient on the teacher side (forward passes only), whereas Seq-KD requires expensive decoding (beam search).
• Architecture: Seq-KD is flexible regarding architecture mismatches; Word-KD requires identical vocabularies.

4. Results & Findings

• Dominance of Seq-KD: Sequence-Level KD is the most popular method, particularly for Low-Resource MT (as data augmentation) and Non-Autoregressive MT (as a regularizer).
• Compression Myth: Only a small fraction of papers use KD strictly for compression. In many cases, KD is used to improve translation quality or adapt models without changing model size.
• Regularization vs. Compression: KD acts as a mode reduction technique, lowering the entropy of the training data. This stabilizes training but can lead to overfitting on frequent patterns, resulting in:
  • Increased hallucination rates.
  • Amplification of gender biases.
  • Poor performance on low-frequency words.
• Evaluation Crisis:
  • Datasets: The field relies heavily on a few datasets (WMT14/15/17), limiting generalizability.
  • Metrics: Over-reliance on BLEU, despite the availability of better metrics like COMET and ChrF.
  • Language Bias: ~80% of research focuses on English-centric translation directions.
• Feature-Based KD: Despite achieving State-of-the-Art (SOTA) results in some recent studies, feature-based methods remain marginal in the MT literature compared to response-based methods.

5. Significance & Future Directions

This survey is significant because it redefines KD in MT from a mere compression tool to a versatile supervision shaping mechanism. It highlights that while KD improves efficiency and quality, it introduces specific risks (bias, hallucination) that must be managed.

Key Recommendations for Future Research:

1. Unified Evaluation: Adopt standard protocols using diverse metrics (COMET, ChrF, rare-word accuracy) and test on both high-resource and low-resource pairs.
2. Diversity Preservation: Actively monitor and mitigate the loss of linguistic diversity caused by KD's mode-reduction effect to prevent bias amplification and hallucination.
3. LLM Integration: Investigate the trade-offs of using LLMs as teachers or data generators, focusing on contamination risks and the quality-quantity balance of synthetic data.
4. Reproducibility: Release distilled datasets, sampling seeds, and checkpoints to enable fair comparison.

The authors conclude that the field must move toward a more unified view of KD applications, addressing the "many roads not taken" (e.g., adversarial KD, graph-based KD) and establishing best practices for safety and fairness.