Technical Summary: KIT's Submission to IWSLT 2026 Long-Form Speech Instruction Following

Problem Statement

The paper addresses the significant gap in current Speech Language Models (SLMs) regarding the effective processing of long-form audio (up to 15 minutes). While recent architectures have integrated pre-trained audio encoders with Large Language Models (LLMs) or developed true multimodal foundation models, most rely on encoders like Whisper that natively support only 30-second inference passes. Newer models (e.g., Phi-4, Qwen2.5-Omni) remove architectural constraints but lack exposure to long-form audio during training, leading to performance degradation even on basic Automatic Speech Recognition (ASR).

Furthermore, the IWSLT 2026 Instruction Following Track introduced a "surprise task" and long-form requirements, challenging models to generalize without overfitting to known tasks or short utterances. Existing datasets are predominantly short-form (<30 seconds), creating a data scarcity for long-form instruction following across multiple languages (German, English, Italian, Chinese) and tasks (ASR, Speech Translation, Spoken QA, Summarization, Audio Chaptering).

Methodology

1. Data Augmentation Framework

To bridge the gap between short-form corpora and long-form requirements, KIT employed a three-stage augmentation pipeline yielding over 1 million training instances across six tasks and four languages:

• Segment Concatenation: Short utterances were grouped by speaker and session to construct long-form audio (up to 15 minutes). Sources included YTSeg, NUTSHELL (ACL talks), EuroParl, and LibriSpeech.
• LLM-Based Label Generation: For datasets lacking task-specific annotations (e.g., NUTSHELL for ASR or SQA), models like gemma-3-12b-it and Qwen3.5-27B generated task labels, questions, and summaries.
• Cross-Lingual Translation: English-only references were translated to German, Italian, and Chinese using translategemma-12b-it, selected for its superior reference-free translation quality.
• Preprocessing: Applied text normalization (removing non-lexical metadata), truecasing, and punctuation restoration using NVIDIA NeMo's BERT-based model.

2. Model Architectures

The submission evaluated two distinct systems:

• End-to-End (Primary): Based on Qwen2.5-Omni-7B, trained with LLamaFactory. This model processes audio and text prompts jointly.
• Cascaded (Contrastive): A two-stage pipeline using parakeet-tdt-0.6b-v2 for ASR followed by Qwen2.5-7B-Instruct for text-based instruction following. The audio field was replaced with a transcript prefix to simulate a text-only input for the second stage.

3. Training Strategies

• Data Interleaving: To address class imbalance (e.g., SQA comprising ~45% of data), the authors compared fixed-probability sampling against temperature-scaled sampling. They found T=2 (sampling proportional to the square root of dataset size) to be the optimal strategy for multimodal speech instruction following, balancing task representation better than fixed heuristics.
• Prompt Engineering: A fixed system prompt was used for global consistency, while task-specific instructions were varied to prevent template memorization.
• Chain-of-Thought (CoT) Conditioning: The authors experimented with task tokens (e.g., <|asr|>, <|ssum|>) to guide task routing but observed a collapse in task discrimination, where the model defaulted to the structurally similar Speech Summarization task.

4. Re-Ranking Strategy

A critical component of the methodology was addressing the "unknown task identity" constraint at inference time. The authors generated 17 candidates per segment (greedy, segmented, and sampled) and evaluated six re-ranking strategies.

• Failure Mode Identified: Likelihood-based re-ranking was found to systematically degrade semantic tasks (SQA, Summarization) by spuriously selecting candidates generated from segmented audio processing (SHAS) rather than holistic inference.
• Proposed Solution: A combined Likelihood + Minimum Bayes Risk (MBR) strategy. MBR acted as a regularizer, suppressing spurious likelihood-driven selections while retaining benefits for ASR. This hybrid approach was applied only to English and Chinese tracks, where it yielded consistent improvements.

Key Results

Performance on MCIF Long Track

• End-to-End vs. Cascaded: The cascaded system excelled in ASR (5.88% WER) and Speech Translation (80.81% COMET) but lagged in semantic tasks (SQA, Summarization) due to error propagation. The end-to-end system outperformed the cascaded approach on semantic tasks but struggled with raw ASR on long-form audio without specific fine-tuning.
• Fine-Tuning Impact: Temperature-scaled sampling (T=2) consistently outperformed fixed-probability sampling. In-domain adaptation on NUTSHELL (scientific talks) and Italian-specific data further improved performance.
• CoT Failure: The task-token conditioning approach resulted in severe degradation (e.g., 79.24% WER for ASR) due to the model's inability to distinguish between similar tasks, collapsing into a single fallback behavior.

Re-Ranking Outcomes

• Likelihood + MBR: This combined strategy achieved the best overall trade-off. It retained significant ASR improvements (reducing WER by ~19 points compared to greedy) while minimizing degradation on SQA and Summarization.
• Positional Bias: Sequential pairwise re-ranking methods exhibited strong positional bias (preferring the first candidate), which Round-Robin and Bracket methods mitigated but did not fully resolve without the MBR regularizer.

Official IWSLT 2026 Results

• Long Track: The contrastive (cascaded) system achieved the strongest results in Speech Translation, Quality Estimation (QE), the surprise task, and ASR. The primary (end-to-end) system performed better on SQA, Summarization, and Audio Chaptering.
• Short Track: The primary system achieved the strongest results in ST, SQA, and ASR.
• Quality Estimation (QE): A notable disparity emerged where the primary (audio-based) submission achieved 0.0 QE accuracy, while the cascaded (transcript-based) submission reached 0.722, highlighting the difficulty of audio-based pipelines on unseen QE tasks.

Significance and Claims

The paper claims five primary contributions to the field of long-form speech instruction following:

1. Data Framework: A reproducible pipeline to convert short-form corpora into 1M+ long-form instances, demonstrating that existing resources can be repurposed via concatenation and normalization.
2. Sampling Strategy: Empirical evidence that temperature-scaled sampling (T=2) is a robust default for balancing multimodal speech tasks, extending prior text-only findings to speech.
3. Negative Result on CoT: A demonstration that prefix-based task routing fails under conditions of task similarity and imbalance, leading to a collapse in task discrimination rather than improved performance.
4. Re-Ranking Analysis: The identification of a specific failure mode where likelihood-based re-ranking harms semantic tasks by favoring segmented candidates, and the proposal of a Likelihood+MBR hybrid to resolve the ASR-vs-semantics trade-off.
5. System Comparison: A systematic evaluation showing that while cascaded systems excel in transcription-heavy tasks, end-to-end models are superior for semantic reasoning, with neither approach consistently dominating all tasks.

The authors conclude that their data augmentation code and model checkpoints are released to support future research, emphasizing that effective long-form processing requires careful balancing of data distribution, task routing, and candidate selection strategies.