NAVER LABS Europe Submission to the Instruction-following 2026 Short Track

Technical Summary: NAVER LABS Europe Submission to the Instruction-following 2026 Short Track

Problem Statement
This paper addresses the challenges of the IWSLT 2026 Instruction-following Speech Processing Short Track under a constrained setting. The objective is to develop a unified system capable of jointly performing Automatic Speech Recognition (ASR), Speech Translation (ST), and Spoken Question Answering (SQA) from English speech input into Chinese, Italian, and German. A key constraint is the reliance on specific foundation models: the SeamlessM4T-v2-large speech encoder and the Qwen3-4B-Instruct-2507 Large Language Model (LLM). The system must also handle a "surprise" instruction-following task revealed only at test time. The authors note that the chosen LLM backbone (Qwen3-4B) is significantly smaller and weaker than models used in previous years, presenting a challenge in maintaining performance while reducing model size and computational requirements.

Methodology
The authors propose a multi-stage training pipeline (illustrated in Figure 1) that trains components in parallel before integrating them:

1. Speech Projector (SpeechMapper): Instead of a standard transformer-based projector, the team employs an updated SpeechMapper. This module maps averaged speech representations from the SeamlessM4T-v2-large encoder into the embedding space of the frozen LLM.

   • Training Objective: The projector is trained using ASR data with a composite loss function comprising:
     • L1 Alignment Loss: Element-wise distance between speech and text embeddings.
     • Cosine Similarity Loss: Angular alignment in the embedding space.
     • Softmax Contrastive Loss: Separation from non-target tokens.
     • CTC Loss: Added at an intermediate layer to stabilize training.
   • Efficiency: This approach requires only the LLM's tokenizer and embedding layer, avoiding full LLM forward passes during projector training, thereby reducing GPU memory and computational costs.

2. Text LoRA Adapters: Parallel to the projector training, Low-Rank Adaptation (LoRA) weights are trained on text-to-text data (Machine Translation and Question Answering) using the same frozen LLM backbone.

3. Multimodal Adaptation: The trained SpeechMapper and LoRA adapters are reloaded and jointly fine-tuned on a mixture of speech and text tasks. The pipeline utilizes a specific prompt format where instructions are formulated as questions in the target language to reduce task confusion (ASR vs. ST) and improve generalization.

4. Data Strategy:

   • Synthetic Data (fakACL): To bridge the domain gap between training data and the evaluation set (scientific presentations), the authors generated fakACL, a synthetic dataset of scientific talks. This was created by prompting the LLM to generate presentation scripts, segmenting them, synthesizing speech via SeamlessM4T-v2-large TTS, and generating associated QA pairs.
   • Data Augmentation: Existing datasets (CoVoST2, EuroParlST, GigaST, LibriSQA) were processed with specific modifications, including MCQ answer expansion (to include full option text rather than just letters) and the creation of "invalid" splits using similarity-based and LLM-based methods to improve robustness.

Key Contributions

• SpeechMapper Integration: The replacement of the previous transformer-based projector with SpeechMapper, which enables learning speech-to-LLM embeddings without requiring LLM forward passes, significantly lowering hardware requirements.
• Domain-Specific Synthetic Data: The introduction of the fakACL dataset, which aligns the training distribution with the evaluation domain (scientific presentations), addressing a known mismatch in previous iterations.
• Efficient Architecture: The development of a system that outperforms the previous year's best submission while relying on a substantially weaker (smaller parameter count) LLM backbone and being more compact.

Results
The system was evaluated on ASR (WER), ST/MT (COMET), and SQA/QA (LLM-as-a-judge accuracy).

• Performance: The final multimodal system (Setup 1) achieved results tied for first place in the overall short track ranking.
• Comparative Analysis: Despite using the smaller Qwen3-4B backbone, the system matched or exceeded the performance of last year's best system (BEST-IWSLT25-IF) which used a larger backbone.
• Trade-offs: The authors observed a trade-off between ASR/ST performance and SQA performance. Setup 1 prioritized SQA (achieving 0.428 on MCIF SQA), while Setup 2 showed stronger ASR/ST results. The final submission prioritized SQA as the primary criterion.
• Baseline Observations: The text-only Qwen3-4B backbone performed significantly worse than SeamlessM4T-v2-large or Llama-3.1-8B in zero-shot settings, particularly for Italian, motivating the heavy use of LoRA fine-tuning and synthetic data up-sampling.

Significance and Claims
The paper claims that the combination of an improved speech projection mechanism (SpeechMapper) and domain-specific synthetic data (fakACL) allows a model with a weaker LLM backbone to outperform the previous year's state-of-the-art system in the constrained short track. The authors emphasize that their approach is "considerably more compact" and relies on fewer computational resources during the projector training phase. The work demonstrates that effective instruction-following in speech processing can be achieved without massive LLM backbones if the speech-to-text alignment and domain adaptation are handled efficiently. The system successfully handles the surprise task and multilingual requirements, validating the pipeline's robustness.