Technical Summary: Qwen3-ASR Technical Report

Problem Statement

The field of Automatic Speech Recognition (ASR) has transitioned from traditional end-to-end paradigms (e.g., Transducer, AED) to Large Audio-Language Model (LALM) approaches. While LALMs offer superior capabilities in long-form transcription, noise robustness, and world-knowledge integration, existing solutions often face limitations in real-world deployment:

1. Benchmark Saturation: Open-source ASR models often show marginal differences on standard public benchmarks despite significant quality gaps in complex, real-world scenarios (e.g., dialects, elderly speech, singing).
2. Fragmented Tooling: Timestamping (forced alignment) is typically a post-processing step using separate tools (e.g., CTC, CIF, MFA) that lack multilingual unification and flexibility in temporal granularity.
3. Efficiency vs. Accuracy Trade-off: There is a need for models that balance high accuracy with low latency for on-device or high-concurrency industrial applications.
4. Specialized Scenarios: Robust recognition of singing voice, songs with background music, and diverse dialects remains challenging for general-purpose ASR systems.

Methodology

1. Model Architecture

The Qwen3-ASR family is built upon the Qwen3-Omni foundation model, leveraging its strong audio understanding capabilities. The architecture consists of:

• AuT Encoder: An attention-encoder-decoder (AED) based encoder pretrained separately. It downsamples Fbank features (100Hz) by 8x to produce 12.5Hz tokens. It utilizes a dynamic flash attention window (1s–8s) to support both streaming and offline inference.
• Projector & LLM: A projector maps encoder outputs to the LLM space.
  • Qwen3-ASR-1.7B: Built on Qwen3-1.7B with a 300M parameter AuT encoder (1024 hidden size).
  • Qwen3-ASR-0.6B: Built on Qwen3-0.6B with a 180M parameter AuT encoder (896 hidden size), optimized for efficiency.
• Qwen3-ForcedAligner-0.6B: A novel non-autoregressive (NAR) model reframing forced alignment as a slot-filling task. It uses a pretrained AuT encoder and the Qwen3-0.6B LLM to predict discrete timestamp indices for special [time] tokens inserted into the transcript.

2. Training Strategies

The training pipeline involves four distinct stages:

1. AuT Pretraining: Trained on ~40 million hours of pseudo-labeled ASR data (mostly Chinese and English) to establish general audio representations.
2. Omni Pretraining: Utilizes the Qwen3-Omni foundation trained on 3 trillion tokens of multi-modal (audio, vision, text) data.
3. ASR Supervised Fine-Tuning (SFT):
   • Uses a smaller, disjoint multilingual dataset.
   • Incorporates non-speech data, streaming-enhancement data, and context biasing.
   • Instruction Mitigation: The model is trained to be ASR-only, ignoring natural language instructions in prompts to prevent instruction injection failures.
   • Output Format: Generates structured outputs indicating language and text (e.g., language English<asr_text>...) or None if no speech is detected.
4. ASR Reinforcement Learning (RL): Employs Group Sequence Policy Optimization (GSPO) on ~50k utterances (including noise, multilingual, and functional data) to enhance noise robustness and transcription stability.

3. Forced Alignment Training

• Pseudo-Labeling: Uses Montreal Forced Aligner (MFA) outputs as pseudo-labels, which are then distilled and smoothed by the model to reduce systematic shifts.
• Causal Training: Unlike standard next-token prediction, the model uses causal training without sequence shifting to explicitly recognize and fill timestamp slots.
• Dynamic Slot Insertion: Randomly inserts start/end timestamp slots during training to improve generalization.

Key Contributions

1. State-of-the-Art All-in-One ASR:

   • Supports 52 languages and dialects (30 languages + 22 Chinese dialects).
   • Achieves SOTA performance among open-source models and competes with top proprietary APIs (GPT-4o, Gemini-2.5-Pro, Doubao-ASR).
   • Demonstrates robustness in complex scenarios: singing voice, songs with background music, elderly/children speech, and extreme noise.

2. Novel Non-Autoregressive Forced Alignment:

   • Introduces Qwen3-ForcedAligner-0.6B, the first LLM-based multilingual forced aligner.
   • Supports flexible granularities (word, sentence, paragraph) across 11 languages.
   • Operates in a non-autoregressive manner, predicting all timestamps simultaneously, significantly improving inference speed.

3. Efficiency and Scalability:

   • Qwen3-ASR-0.6B achieves an average Time-to-First-Token (TTFT) as low as 92ms and a throughput of 2,000 seconds of audio per second at 128 concurrency.
   • Supports unified offline and streaming inference with a maximum single-inference duration of 1,200 seconds (20 minutes).

4. Open-Source Ecosystem:

   • Releases weights for three models under the Apache 2.0 license.
   • Provides a comprehensive inference and fine-tuning framework supporting multi-granularity alignment, streaming, and multilingual processing.

Experimental Results

ASR Performance

• Public Benchmarks: On English and Chinese benchmarks (LibriSpeech, WenetSpeech, Fleurs, etc.), Qwen3-ASR-1.7B consistently outperforms open-source baselines (Whisper-large-v3, FunASR) and remains competitive with commercial APIs.
• Internal Robustness Suite: In stress tests involving 16 English accents, 22 Chinese dialects, and difficult conditions (elderly speech, tongue twisters, extreme noise), Qwen3-ASR-1.7B achieved the lowest Word Error Rates (WER) across all subsets, surpassing commercial APIs.
• Multilingual & LID: The models achieve high accuracy in Language Identification (LID) (e.g., 98.7% on Fleurs for 1.7B) and maintain strong performance across 30 languages.
• Singing Voice: Qwen3-ASR-1.7B achieves the best results on singing-only benchmarks (M4Singer, MIR-1k-vocal) and significantly outperforms baselines on long-form song transcription with background music, where other models often fail.

Forced Alignment Performance

• Accuracy: Qwen3-ForcedAligner-0.6B reduces the Accumulated Average Shift (AAS) by 67%–77% compared to existing methods (Monotonic-Aligner, NFA, WhisperX) on human-labeled datasets.
• Robustness: Unlike baselines that degrade on long utterances, the proposed model maintains low AAS even on 300-second concatenated clips and cross-lingual scenarios.
• Efficiency: Achieves a Real-Time Factor (RTF) close to 0.001, processing 1,000 seconds of audio per second.

Significance and Claims

The paper claims that the Qwen3-ASR family represents a significant step forward in making high-performance, multilingual, and robust speech recognition accessible to the community. By leveraging the LALM paradigm, the models address challenges that traditional ASR struggles with, such as long-form transcription, dialectal variation, and singing voice recognition.

The introduction of Qwen3-ForcedAligner-0.6B is highlighted as a critical functional component, filling a gap in the speech technology stack by providing a unified, multilingual, and efficient solution for timestamping that outperforms specialized tools in both accuracy and versatility. The authors emphasize that the release of these models under an open license, accompanied by a reproducible framework, aims to accelerate research and development in automatic speech recognition and audio understanding.