Reliability Gated Multi-Teacher Distillation for Low Resource Abstractive Summarization

1. Problem Statement

Large sequence-to-sequence models have revolutionized abstractive summarization but are often too computationally expensive for low-resource settings. Knowledge Distillation (KD) is a standard solution to transfer knowledge from large "teacher" models to smaller "student" models. However, existing KD approaches face two critical challenges in generative tasks:

• Teacher Instability: In multi-teacher setups, teachers often disagree on specific tokens (e.g., entities or phrasing). Naively averaging their predictions introduces noisy supervision that can degrade student performance.
• Capacity Mismatch: Standard KD often treats all teachers equally, ignoring the fact that a student model's representational capacity may align better with a smaller teacher than a much larger one.
• Low-Resource Constraints: In low-resource languages (like Bangla), data scarcity makes every supervision signal valuable, yet noisy signals from unreliable teachers can be detrimental.

The paper asks: When does multi-teacher supervision improve summarization, and when does it introduce noise?

2. Methodology

The authors propose a framework that combines a heterogeneous teacher ensemble with two novel, reliability-aware training components: EWAD and CPDP.

A. Framework Overview

• Input Routing: Documents are routed based on length. Short documents ($\le$ 1024 tokens) go to the KD branch; longer documents use a MapReduce pipeline (chunking, summarizing, and deduplicating) to handle context window limits.
• Teacher Ensemble: The system utilizes a mix of same-vocabulary logit-level KD (e.g., BanglaT5 $\to$ BanglaT5-small) and cross-architecture pseudo-label KD (e.g., mT5 $\to$ BanglaT5) to create a heterogeneous teacher set.

B. Core Innovations

1. EWAD (Entropy-Weighted Agreement-Aware Distillation)
EWAD is a token-level mechanism that dynamically routes supervision between teacher distillation and gold supervision based on two axes:

• Confidence: Measures how decisive a teacher is (via normalized entropy).
• Agreement: Measures the divergence between teachers (via Jensen-Shannon Divergence).
• Mechanism:
  • If teachers are both confident and agree ($\lambda_t \approx 1$), the student receives weighted soft-label KD.
  • If teachers disagree or are uncertain ($\lambda_t \approx 0$), the loss automatically falls back to gold-label cross-entropy, preventing the student from learning from contradictory signals.
  • Formula: $L_{EWAD} = \frac{1}{T} \sum [\lambda_t L_{KD}^t + (1-\lambda_t) L_{CE}^t]$.

2. CPDP (Capacity-Proportional Divergence Preservation)
CPDP is a geometric regularizer that constrains the student's position in distribution space relative to teachers of different capacities.

• Concept: In a heterogeneous setup (e.g., 32B and 14B teachers distilling to a 3B student), the student should be geometrically closer to the smaller teacher (14B) than the larger one (32B).
• Mechanism: It penalizes deviations from a target divergence ratio defined by the inter-teacher divergence. It ensures the student does not collapse toward one teacher arbitrarily but maintains a structural relationship consistent with the capacity hierarchy.

C. Additional Techniques

• Confidence-Adaptive Temperature: The distillation temperature ($\tau$) is adjusted per sample based on teacher entropy (sharper for confident teachers, softer for uncertain ones).
• Cross-Lingual Pseudo-Label KD: An offline pipeline where a large teacher generates pseudo-summaries for ten languages, allowing a small student to train 100x faster than online KD.

3. Key Contributions

1. EWAD: A reliability-gated objective that dynamically switches between teacher and gold supervision based on inter-teacher agreement, preventing noise injection.
2. CPDP: A novel geometric regularizer that enforces capacity-proportional divergence, ensuring the student's distribution aligns with the teacher hierarchy.
3. Empirical Insights: Demonstrated that logit-level KD is the most reliable driver of performance, while complex multi-component KD is highly dependent on output length (helping short summaries but hurting long ones).
4. Cross-Lingual Validation: Showed that the framework generalizes across ten typologically diverse languages, retaining 71–122% of teacher ROUGE-L scores at 3.2x compression.
5. Evaluation Critique: Revealed significant calibration bias in single-judge LLM evaluations, advocating for multi-judge pipelines.

4. Experimental Results

A. BanglaT5 Experiments (Low-Resource)

• Datasets: BTS (short summaries, ~8 tokens) and BanSum (long summaries, ~106–248 tokens).
• Findings:
  • Logit-level KD (A2) provided the most consistent gains, retaining ~93–96% of teacher ROUGE-L.
  • Complexity vs. Length: Adding EWAD/CPDP components improved semantic similarity for short summaries (BTS) but degraded performance on long summaries (BanSum). The authors attribute this to noise accumulation in longer autoregressive sequences.
  • Data Scaling: Increasing training data from 20K to 141K samples yielded larger improvements than any loss engineering (EWAD/CPDP).

B. Qwen-2.5 Experiments (High-Capacity)

• Setup: Distilling 32B/14B teachers into a 3B student.
• Findings:
  • Direct Fine-Tuning Outperformed KD: The baseline (direct fine-tuning without distillation) achieved the highest ROUGE and BERTScore.
  • Capacity Ceiling: When the student model is already close to the teacher's performance, reliability-aware routing (EWAD/CPDP) introduced variance without providing cleaner supervision, failing to surpass the baseline.

C. Cross-Lingual Results

• The offline pseudo-label KD pipeline successfully transferred knowledge across 10 languages (Hindi, Urdu, Persian, etc.).
• Performance: The student (300M) retained 71–93% of teacher ROUGE-L on most languages and surpassed the teacher (122% retention) on Pashto.

D. Evaluation Analysis

• LLM Bias: A human-validated multi-judge study showed that single-judge pipelines (e.g., GPT-5.2) suffer from calibration bias, assigning high faithfulness scores to hallucinated content. Multi-judge evaluation was proven necessary for reliable assessment.

5. Significance and Conclusion

The paper challenges the assumption that more complex distillation objectives always yield better results. Its primary significance lies in the following conclusions:

1. Simplicity Wins: For low-resource summarization, simple logit-level KD often outperforms complex, multi-component distillation.
2. Context Matters: The effectiveness of multi-teacher supervision is strictly modulated by output length and the capacity gap between student and teacher.
3. Data > Loss Engineering: In current model scales, increasing data coverage is more effective than designing increasingly complex loss functions.
4. Reliability is Key: The proposed "reliability-gated" framework provides a crucial lens for understanding when teacher supervision is beneficial versus when it acts as noise.

Ultimately, the work suggests that future progress in generative KD depends less on complex loss engineering and more on understanding the conditions under which teacher supervision is genuinely reliable.