Technical Summary: FBK's Long-form SpeechLLMs for IWSLT 2026 Instruction Following

Problem Statement

The paper addresses the challenge of extending Speech Large Language Models (SpeechLLMs) from short-form to long-form instruction following under constrained settings. While SpeechLLMs have successfully integrated speech encoders with LLM decoders for tasks like Automatic Speech Recognition (ASR), Speech Translation (ST), Spoken Question Answering (SQA), and Speech Summarization (SSUM), long-form processing introduces significant hurdles. These include high computational costs, long acoustic token sequences, the need for discourse-level context modeling, and a heightened risk of hallucination. Previous work in IWSLT 2025 highlighted the scarcity of participants in the long track and the instability of generating long-form outputs. The authors aim to determine how effectively short-form SpeechLLMs can be extended to long-form settings, which segmentation strategies are most effective, and how to reliably evaluate these systems given the prevalence of hallucinations.

Methodology

Model Architecture

The system is built upon two pretrained models: SEAMLESSM4T-V2-LARGE as the speech encoder and QWEN3-4B-INSTRUCT as the LLM decoder.

• Speech Encoder: Processes 16 kHz audio into 80-dimensional log-Mel filterbank features, which are downsampled to 50 Hz. It utilizes a 24-layer Conformer stack with relative positional representations and depthwise convolutions to capture long-range dependencies and local acoustic structures.
• Modality Adapter: Compresses the 50 Hz speech representations by a factor of 8 (yielding ~6.25 Hz) using inherited convolutional layers and a gated linear unit, followed by a linear projection to align the 1024-dimensional speech features with the 2560-dimensional LLM hidden space.
• LLM Decoder: Leverages the instruction-following capabilities of Qwen3-4B. Speech representations are prepended to text embeddings using a fusion mechanism. The model employs Parameter-Efficient Fine-Tuning (LoRA) on query, key, and output projection modules, updating approximately 112.2M parameters out of a total 4.73B.

Training Strategy

• Short Track: Trained on a unified multilingual dataset comprising CoVoST2, EuroParlST, GigaST, and LibriSQA (totaling ~8,436 hours). Synthetic data augmentation was performed using SEAMLESSM4T and Qwen3-4B to fill annotation gaps (e.g., missing translations or QA pairs) and introduce unanswerable cases for robustness.
• Long Track: Due to the lack of reliable long-form supervision for ASR and ST in existing datasets (Nutshell and YTSeg), the authors constructed long-form training data by artificially concatenating short-form utterances from LibriSQA based on chapter structures. This included inserting silence between utterances and processing transcriptions to maintain sentence boundaries. The training corpus totaled ~9,496 hours.
• Optimization: The model was trained using AdamW with a dynamic bucket sampler to handle variable-length inputs efficiently. For the long track, speed perturbation was disabled, and gradient accumulation was set to 8.

Speech Segmentation Strategies

To handle long-form inference, three segmentation approaches were explored:

1. Fixed-Window: Non-overlapping segments of 15, 30, 45, or 60 seconds.
2. CRDNN-based: Voice Activity Detection (VAD) using the SpeechBrain CRDNN model with varying thresholds to control speech onset/offset detection.
3. Hybrid: A combination of pause-based segmentation and duration constraints, utilizing a divide-and-conquer algorithm to recursively split segments exceeding the maximum duration at the longest pause.

Key Contributions

1. Practical Solutions: The paper presents effective SpeechLLM solutions for both short-form and long-form instruction following under constrained settings, utilizing specific pretrained architectures.
2. Empirical Segmentation Analysis: It provides a comparative evaluation of three distinct speech segmentation strategies (fixed-window, CRDNN-based, and hybrid) for long-form inference.
3. Hallucination-Aware Evaluation: The study introduces the Hallucination-Penalized Instruction-Following Score (HIFS) to account for unstable long-form generation. It details how hallucinations manifest primarily as repetitive insertions, which severely degrade ASR and SSUM metrics.
4. Capability Retention Analysis: The work demonstrates that short-form capabilities are largely retained even after extending the model to long-form settings, provided the training data is appropriately constructed.

Results

Short-Form Performance

The primary short-form system achieved a Short-form Instruction-Following Score (SIFS) of 2.0708 on the MCIF benchmark. This comprised:

• ASR Accuracy: 0.8877
• Average ST COMET Score: 0.7550
• Average SQA BERTScore: 0.4281

Long-Form Performance

The evaluation of long-form systems revealed that segmentation strategy significantly impacts performance:

• Fixed 30-Second Segmentation: Achieved the highest HIFS score of 2.0663, demonstrating the most robust performance across ASR, ST, SQA, and SSUM. It offered the best trade-off between preserving context and avoiding unstable generation.
• CRDNN Segmentation: The best configuration (0.5, 0.25, 0.5) yielded an HIFS of 1.9454. This method was highly sensitive to parameter settings; relaxing thresholds improved SQA and SSUM but degraded ST performance.
• Hybrid Segmentation: The best hybrid configuration achieved an HIFS of 2.0139, outperforming CRDNN but remaining below the fixed 30-second window.

Hallucination Analysis

Hallucinations were found to manifest predominantly as repetitive insertions.

• In the CRDNN strategy, including hallucinated outputs increased ASR Word Error Rate (WER) from 0.1579 to 1.0158, driven by a surge in insertion errors (from 271 to 13,969).
• SQA was less affected by hallucinations compared to ASR and SSUM.
• The study applied a post-editing strategy using regular expressions to remove repetitive insertions, improving the reliability of long-form outputs before evaluation.

Significance and Claims

The paper claims that fixed 30-second segmentation provides the most robust strategy for long-form inference, outperforming adaptive segmentation methods that require careful tuning. It highlights that while short-form capabilities are largely preserved during long-form extension, the primary challenge remains the mitigation of hallucinations, which disproportionately affect generation-heavy tasks like ASR and SSUM.

The authors modestly conclude that while their systems achieve competitive results, limitations exist. These include the completion of only one training epoch due to time constraints, the potential degradation of short-form performance due to using a subset of short-form data for long-form training, and the reliance on surface-level regular expressions for hallucination mitigation, which may miss subtler semantic inconsistencies. The work underscores the difficulty of extending short-form SpeechLLMs to long-form contexts and identifies better segmentation and hallucination mitigation as critical future directions.