💬 NLP

NAVER LABS System Re-implementation for the IWSLT 2026 Instruction-Following Task

This paper presents a re-implementation of the NAVER LABS IWSLT 2025 instruction-following pipeline for the 2026 Shared Task, adapting it with mandated SeamlessM4T-v2-large and Qwen3-4B-Instruct components while introducing 100k synthetic training examples to achieve strong performance on speech translation and spoken question answering benchmarks.

Original authors: Anand Kamble, Aniket Tathe

Published 2026-07-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Anand Kamble, Aniket Tathe

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read librarian (the LLM, or Large Language Model) who knows how to write and answer questions perfectly. However, this librarian is "deaf"—they can't hear audio files, only read text.

On the other hand, you have a specialized audio recorder (the Speech Encoder) that is excellent at listening to sound and turning it into a digital "sound map," but it doesn't know how to speak or reason.

The goal of this paper is to build a translator bridge between the deaf librarian and the audio recorder so they can work together as a single team. This team is designed to follow instructions like "Translate this speech," "Summarize this audio," or "Answer questions about what you heard."

Here is how the authors built this team, using simple analogies:

1. The Team Members

The Librarian (Qwen3-4B): A powerful AI that already knows how to follow instructions in text.
The Recorder (SeamlessM4T-v2-large): A state-of-the-art tool that converts human speech into digital data.
The Bridge (Projector): A new, trainable component that translates the "sound map" from the recorder into a language the librarian can understand.

2. The Three-Stage Training Process

The authors didn't just glue these two together; they trained them in three specific steps, like training a new employee:

Stage 1: Learning to Listen (Projector Alignment)
- What happened: The librarian and the recorder were frozen (locked in place). Only the "Bridge" was trained.
- The Analogy: Imagine the librarian is sitting in a soundproof room. The Bridge learns to take the raw sound waves from the recorder and convert them into written notes that the librarian can read. They practiced with thousands of examples of speech and text (like transcripts of speeches) so the Bridge learns the right "vocabulary" to describe sounds.
Stage 2: The Librarian's Text Boot Camp (Text-Only LoRA)
- What happened: The audio was turned off. The librarian was given a massive amount of text data to practice answering questions and translating languages, but this time, they were allowed to make small, efficient adjustments to their brain (using a technique called LoRA).
- The Analogy: The librarian is now in a quiet library, practicing their translation and question-answering skills on paper. They aren't listening to audio yet; they are just sharpening their logic and language skills so they are ready for the real world. The authors tested different "adjustment sizes" (ranks) to see which one made the librarian smartest without breaking their memory.
Stage 3: The Full Rehearsal (Multimodal Merge)
- What happened: The audio was turned back on. Now, the Bridge and the Librarian practiced together.
- The Analogy: The librarian and the recorder are finally in the same room. They practice listening to a speech, having the Bridge translate the sound, and the librarian responding. To make sure the librarian doesn't forget their text skills while learning to listen, they alternate between listening exercises and text-only exercises. This ensures they don't get "confused" (a problem called catastrophic forgetting).

3. The "Fake" Practice Data

One of the paper's unique contributions is creating 100,000 synthetic examples.

The Analogy: Since real-world data for every possible task (like "describe the speaker's voice tone" or "extract keywords") was scarce, the authors used another AI (Gemma) to invent 10,000 fake practice scenarios for 10 different types of tasks.
They created fake transcripts and fake audio descriptions to train the team on things like:
- Keyword Extraction: Pulling out the most important words.
- Voice Description: Describing if a speaker sounds "confident," "slow," or "loud" just by hearing them.
- Summarization: Condensing a long speech into one sentence.

4. The Results

When they tested their final team on the official "exam" (the MCIF benchmark):

Translation: They became very good at translating speech from English to Chinese, German, and Italian.
Question Answering: They improved significantly at answering questions based on what they heard in English.
The Trade-off: Interestingly, while they got better at understanding the meaning of speech (translation and questions), their ability to write down the exact words heard (transcription) got slightly worse compared to the raw recorder. This is a common trade-off when teaching a system to understand "meaning" rather than just "sounds."

Summary

The paper is essentially a "how-to" guide for building a multilingual, audio-aware AI assistant. They took a deaf text-expert and a hearing-but-dumb recorder, built a custom bridge between them, trained them in three careful stages, and fed them a massive amount of practice data (both real and AI-generated). The result is a system that can listen to short audio clips and follow complex instructions like a human would.

Limitations Mentioned:

The system currently struggles with audio longer than 15 seconds (it runs out of "mental energy" or memory).
For non-English questions, they had to use machine-translated practice data, which might introduce some "noise" or errors.

Technical Summary: NAVER LABS System Re-implementation for the IWSLT 2026 Instruction-Following Task

Problem Statement

The paper addresses the challenge of developing unified multimodal speech Large Language Models (LLMs) capable of instruction-following across diverse tasks, including Automatic Speech Recognition (ASR), multilingual Speech Translation (ST), and Spoken Question Answering (SQA). While previous systems like the NAVER LABS 2025 submission (Lee et al., 2025) demonstrated competitive performance, they were not publicly released. Furthermore, the IWSLT 2026 Shared Task introduces specific constraints requiring the use of SeamlessM4T-v2-large as the speech encoder and Qwen3-4B-Instruct as the LLM backbone, replacing the components used in the 2025 iteration (LLaMA-3.1-8B). The goal is to provide the first open-source re-implementation of this pipeline adapted to these new constraints and to enhance performance through synthetic data generation and hyperparameter ablation.

Methodology

Architecture

The system follows a three-stage training pipeline, coupling a frozen speech encoder with an instruction-tuned LLM via a lightweight connector:

Speech Encoder: A frozen SeamlessM4T-v2-large encoder produces 1024-dimensional frame representations.
Projector: A trainable module downsamples frames by a factor of 3 via averaging, processes them through a 4-layer Transformer encoder, and projects the output to the LLM's hidden size.
LLM Backbone: A Qwen3-4B-Instruct model adapted with LoRA (Low-Rank Adaptation). Speech embeddings are prepended at a <|speech|> placeholder.

Training Pipeline

The training process is divided into three distinct stages:

Stage 1: Projector Alignment
- The encoder and LLM are frozen; only the projector is trained.
- Data: CoVoST 2 and EuroParlST for ASR and ST tasks.
- Configurations: Two variants are explored:
  - A.1: Focuses on ASR and ST (EN→DE, IT, ZH).
  - A.2: Adds LibriSQA for SQA tasks.
- Hyperparameters: 4 epochs, constant learning rate ( $1\times10^{-4}$ ), AdamW optimizer.
Stage 2: Text-Only LoRA Pre-training
- The projector is frozen; the LLM is adapted via LoRA without audio input.
- Data: Machine-translated transcripts from CoVoST 2/EuroParlST (60%) and LibriSQA (40%).
- Ablation: Three LoRA configurations (V1, V2, V3) are tested varying rank ( $r$ $r$ ), $\alpha$ $α$ , learning rate, and layer selection (attn+FF vs. all-linear).
  - V1: $r=8, \alpha=16, lr=3\times10^{-4}$ (attn+FF).
  - V2: $r=16, \alpha=32, lr=1\times10^{-5}$ (all-linear, cosine decay).
  - V3: $r=32, \alpha=64, lr=2\times10^{-4}$ (all-linear, cosine decay).
Stage 3: Multimodal Merging
- Both the Stage 1 projector and the selected Stage 2 LoRA adapters are jointly fine-tuned; the encoder remains frozen.
- Data: A balanced mix of ASR, ST, SQA (EN and multilingual), Speech-to-Text Summarization (NUTSHELL), and Audio Chapter Detection (YTSeg).
- Strategy: To prevent catastrophic forgetting, each speech batch is immediately followed by a paired text-only batch.
- Hyperparameters: 2 epochs on 4×H100 GPUs. Projector uses constant $lr=1\times10^{-5}$ ; LoRA uses cosine decay $lr=3\times10^{-4}$ .

Synthetic Data Construction

To support Stage 3 fine-tuning, the authors constructed 100k synthetic instruction-following examples (10k per task) using open-weight Gemma models:

Text-Grounded Tasks (T1–T7): Generated from reference transcripts using Gemma-4-31B. Tasks include keyword extraction, Named Entity Recognition (NER), gist summarization (EN, DE, ZH), topic labeling, and numeric QA.
Audio-Grounded Tasks (T8–T10): Generated directly from audio using Gemma-4-E4B-it. Tasks involve describing vocal style, tone, and delivery in EN, DE, and ZH.

Key Results

Main Evaluation (MCIF Benchmark)

Evaluated on the constrained, short audio track of the IWSLT 2026 Shared Task:

Speech Translation (ST): The Stage 3 model (A.1 Projector + V1 LoRA) achieved a COMET score of 0.781 for EN–ZH, outperforming the base SeamlessM4T-v2-large (0.638) and the Stage 1 projector-only baselines.
Spoken Question Answering (SQA): The model achieved a BERTScore-F1 of 0.346 on English SQA, a significant improvement over the base model (0.153) and Stage 1 A.1 (0.186).
ASR: While Stage 1 A.2 showed degraded Word Error Rate (WER) compared to the base, Stage 3 recovered performance to 23.49 WER, surpassing the base model's 21.49 but remaining competitive within the pipeline context.

Ablation Studies

LoRA Configuration: In Stage 2 text-only evaluation, V1 (rank 8) yielded the best SQA F1 scores (0.652) compared to V2 (0.650) and V3 (0.600), despite V3 achieving higher Machine Translation BLEU scores. Consequently, V1 was selected for the final Stage 3 merge.
Data Impact: Incorporating SQA data in Stage 1 (A.2) boosted English SQA performance but degraded ASR metrics, highlighting a trade-off managed by the final Stage 3 joint fine-tuning.

Significance and Contributions

The paper claims the following contributions:

Open-Source Reproduction: It provides the first open-source re-implementation of the NAVER LABS instruction-following pipeline, adapted specifically for the IWSLT 2026 constraints (SeamlessM4T-v2-large and Qwen3-4B-Instruct).
Synthetic Dataset: The release of 100k synthetic instruction-following examples across ten speech-centric task types, designed to facilitate further fine-tuning and future research.
Empirical Validation: The study validates that the three-stage approach (projector alignment, text-only LoRA pre-training, multimodal merging) remains effective when transitioning to new model backbones, achieving strong performance in EN–ZH translation and English SQA.
Future Utility: The authors suggest the released data serves as a foundation for richer Stage 3 fine-tuning or future reinforcement learning approaches using task-specific rewards and LLM-as-judge criteria.

Limitations

The authors acknowledge several constraints:

Audio Length: Audio longer than 15 seconds was excluded due to GPU memory limitations, potentially affecting performance on longer utterances.
Multilingual SQA: Cross-lingual SQA relies on machine-translated QA pairs, which may introduce noise.
Evaluation Scope: Stage 2 ablation results are based on text-only subsets and are not directly comparable to the full MCIF benchmark results.