💬 NLP

DuDi: Dual-Signal Distillation with Cross-Lingual Verbalizer

The paper introduces DuDi, a dual-signal distillation framework enhanced by a cross-lingual verbalizer that effectively improves the multilingual capabilities of small language models, particularly for Southeast Asian languages, by combining sequence-level and token-level supervision signals.

Original authors: Patomporn Payoungkhamdee, Tinnakit Udsa, Jian Gang Ngui, Sarana Nutanong, Alham Fikri Aji, Peerat Limkonchotiwat

Published 2026-06-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Patomporn Payoungkhamdee, Tinnakit Udsa, Jian Gang Ngui, Sarana Nutanong, Alham Fikri Aji, Peerat Limkonchotiwat

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Tiny Brain" Struggle

Imagine you have a brilliant, well-read professor (a large AI model) who speaks many languages fluently. Now, imagine you need to teach a small, energetic child (a Small Language Model or SLM) to do the same job.

The problem is that when you shrink a model down to make it faster and cheaper to run, it often loses its ability to speak languages other than English. This is especially true for Southeast Asian (SEA) languages like Thai, Vietnamese, and Indonesian. The "child" model gets confused and makes mistakes, while the "professor" knows the answers perfectly.

The Solution: DuDi (The Super Tutor)

The authors created a new teaching method called DuDi. Think of DuDi not just as a teacher, but as a super-intelligent tutor that uses three specific tricks to help the small model learn faster and better than before.

1. The "Big Picture" Check (Sequence Signal)

The Analogy: Imagine a student writing an essay. A normal teacher might just check if the spelling is right word-by-word. But a great teacher also looks at the whole essay to see if the story makes sense from start to finish.
How DuDi does it: DuDi checks the student's entire response at once. It asks, "Does this whole paragraph look like the correct answer?" This helps the student understand the overall flow and direction, not just individual words.

2. The "Two-Track" Feedback (Dual-Signal Distillation)

The Analogy: Think of learning to ride a bike.
- Track A (Off-Policy): You look at a photo of a perfect rider (the teacher's data) and try to copy it exactly. This gives you a solid foundation.
- Track B (On-Policy): You actually get on the bike and try to ride. When you wobble, the teacher jumps in and says, "Hey, you leaned too far left there!" This helps you correct your own mistakes in real-time.
How DuDi does it: Most methods only do one or the other. DuDi does both. It uses the teacher's perfect data to set the standard, and it watches the student generate its own answers to correct specific mistakes as they happen. This "dual-signal" approach keeps the student on the right track.

3. The "Universal Translator" (Cross-Lingual Verbalizer)

The Analogy: Imagine the teacher speaks Thai, and the student is trying to learn Vietnamese. If the teacher just speaks Thai, the student might get confused about how to say things in Vietnamese.
How DuDi does it: DuDi uses a special "translator prompt."
- The teacher sees a question in Thai and a perfect answer in Thai.
- But, the teacher is asked to explain the answer as if it were being taught in Vietnamese (or English, or any other language in the mix).
- The student then has to generate the answer in that target language.
- Why this works: It forces the student to learn the logic of the answer, not just memorize the words. It's like learning the concept of "January" rather than just memorizing the Thai word for it. This helps the student transfer knowledge between different languages much better.

The Results: A Smaller Model That Punches Above Its Weight

The researchers tested this method on models ranging from very small (0.5 billion parameters) to medium-sized (1.5 billion). They compared DuDi against other popular teaching methods.

The Outcome: DuDi consistently beat the other methods.
The Proof: In a "report card" called SEA-HELM (which tests how well models handle Southeast Asian languages), the DuDi-trained models got the highest scores.
The Surprise: Even when the "professor" (teacher) wasn't perfect at a specific language, DuDi still managed to teach the "student" to do better than it would have on its own.

The "Secret Sauce" Analysis

The authors ran tests to see which part of DuDi was the most important:

Removing the "Big Picture" check: The model got slightly worse.
Removing the "Two-Track" feedback: The model got much worse. This proved that seeing both the teacher's data and the student's own mistakes is crucial.
Removing the "Universal Translator": The model got worse, proving that translating the teaching style across languages is a key factor in success.

Summary

DuDi is a new way to train small AI models to speak Southeast Asian languages. Instead of just copying a big model, it uses a combination of whole-response checks, real-time mistake correction, and a cross-language teaching style to help small models learn faster and smarter. The result is a tiny, efficient AI that can understand and speak multiple languages almost as well as much larger, more expensive ones.

Technical Summary: DuDi – Dual-Signal Distillation with Cross-Lingual Verbalizer

Problem Statement

Small Language Models (SLMs) offer significant advantages in efficiency, scalability, and deployment on resource-constrained edge devices. However, their multilingual capabilities, particularly for Southeast Asian (SEA) languages, degrade severely when model sizes fall below the billion-parameter scale. Standard Supervised Fine-Tuning (SFT) often results in sharp performance drops for sub-billion models (e.g., Qwen2.5-0.5B) compared to their larger counterparts. While Knowledge Distillation (KD) is a common strategy to transfer knowledge from larger teachers to smaller students, existing multilingual KD approaches are often limited to task-specific settings or fail to effectively unify sequence-level and token-level supervision. Furthermore, standard distillation methods often struggle with the training-inference mismatch and the linguistic diversity inherent in SEA languages, where high-quality training data is scarce.

Methodology: The DuDi Framework

The authors propose DuDi (Dual-Signal Distillation with Cross-Lingual Verbalizer), a unified framework designed to enhance multilingual SLMs by integrating three core components:

1. Sequence-Level Signal

DuDi incorporates an online sequence-level objective inspired by Self-Play Fine-Tuning (SPIN). Unlike static iterative methods, DuDi samples responses ( $y' \sim \pi_\theta(x)$ ) in real-time. The model is trained to differentiate its current policy from the ground-truth by maximizing the relative likelihood of the ground-truth responses. This guides the student policy toward the ground-truth direction dynamically.

2. Token-Level Signals (Dual-Signal)

To address the limitations of relying solely on student-generated data or static datasets, DuDi employs a dual-signal approach for token-level supervision:

Off-Policy Signal: Derived from the training corpus, this signal uses teacher-provided logits to ensure the supervisory signal remains strictly grounded in the ground-truth distribution. It prevents the learned policy from drifting away from the target data distribution.
On-Policy Signal: Derived from student-generated responses ( $\tilde{y} \sim \pi_\theta(x)$ ), this signal allows the teacher to refine the student's self-generated outputs, correcting mistakes in real-time.

3. Cross-Lingual Verbalizer

To improve knowledge transferability in multilingual settings, DuDi introduces a cross-lingual verbalizer. This component transforms an input prompt ( $x$ ), ground-truth ( $y$ ), source language ( $l$ ), and a sampled target language ( $l_z$ ) into a structured verbalized prompt ( $z$ ) for the teacher and a corresponding prompt ( $p_z$ ) for the student.

Mechanism: The teacher receives the prompt in the source language with the ground-truth as a demonstration. The student is then instructed to generate a response in the target language (sampled uniformly from the training set, excluding the source language, and including English).
Objective: This forces the student to align its on-policy generation with the teacher's guidance across language boundaries, facilitating richer learning signals.

Unified Training Objective

The final learning objective ( $L_{DuDi}$ ) is a weighted sum of:

Sequence-level alignment (SPIN-style).
Off-policy token-level distillation (Ground-truth guided).
On-policy token-level distillation (Teacher-guided via cross-lingual verbalizer).

Key Contributions

DuDi Framework: A general-purpose multilingual distillation framework that jointly optimizes sequence-level objectives and dual-signal (off-policy and on-policy) token-level supervision.
Cross-Lingual Verbalizer: A novel mechanism that refines teacher feedback by converting inputs and demonstrations into prompts that guide the student to generate responses in different target languages, thereby enhancing transferability.
Comprehensive Analysis: Extensive ablation studies and analyses confirming that sequence-level optimization, token-level supervision, and cross-lingual verbalization provide complementary and transferable learning signals.

Experimental Results

The authors evaluated DuDi using the SEA-HELM benchmark across seven Southeast Asian languages (Indonesian, Vietnamese, Thai, Tamil, Tagalog, Malay, and Burmese).

Performance: DuDi consistently outperformed competitive baselines, including SFT, DFT, SPIN, SDFT, SeqKD, and GKD.
- On Qwen2.5-0.5B, DuDi achieved an average score of 10.1, statistically significantly outperforming the second-best baseline (DFT at 9.7) and SPIN (9.5).
- DuDi demonstrated robustness across different model families (Qwen2.5, Qwen3, Llama3.2) and scales (0.5B to 1.5B students).
- In the Qwen3-0.6B setting, DuDi improved average performance by 15.6% over the strong GKD baseline.
Ablation Studies:
- Removing the off-policy KD caused the largest performance drop (-24.6%), highlighting its critical role in grounding the model in the target distribution.
- Removing the on-policy KD resulted in a -6.6% drop.
- Removing the verbalizer caused a -5.0% drop.
- Removing English from the target language set in the verbalizer degraded performance, suggesting English facilitates better cross-lingual transfer.
Verbalizer Analysis: Overlap ratio analysis showed that the cross-lingual verbalizer achieved the highest agreement between teacher and student logits during on-policy rollouts, confirming it provides the most informative supervision signals.
Initialization: The study confirmed that "cold-start" SFT (initializing from a model fine-tuned on ground-truth data) is crucial; initializing directly from base models or teacher-generated outputs resulted in significantly lower performance.

Significance and Claims

The paper claims that DuDi addresses the specific challenge of scaling down multilingual capabilities by providing a unified framework that leverages complementary learning signals. The authors assert that:

Joint Optimization is Necessary: The components (sequence signal, dual token signals, and verbalizer) are not independently effective but require joint integration to achieve state-of-the-art results in multilingual SLMs.
Cross-Lingual Transfer: The cross-lingual verbalizer is a key innovation that improves teacher-student knowledge transferability, overcoming limitations of English-only or monolingual verbalizers.
Robustness: The framework is robust across different model architectures and scales, making it a viable solution for deploying efficient SLMs in diverse, low-resource linguistic environments like Southeast Asia.

The authors conclude that DuDi offers an effective fine-tuning framework for SLMs in multilingual environments, with all artifacts (code, datasets, models) to be released for open research.