Polyglot-Lion: Efficient Multilingual ASR for Singapore via Balanced Fine-Tuning of Qwen3-ASR

1. Problem Statement

Singapore presents a uniquely challenging environment for Automatic Speech Recognition (ASR) due to its four official languages (English, Mandarin, Tamil, and Malay) coexisting in daily communication, often within single utterances (code-switching). Additionally, the prevalence of "Singlish" and diverse accents complicates recognition.

Existing solutions face significant trade-offs:

• General-purpose models (e.g., Whisper, MMS) suffer from degraded accuracy on low-resource languages (Tamil, Malay) and regional accents.
• Large Audio-Language Models (ALMs) (e.g., Qwen2.5-Omni, SeaLLMs) are expensive to fine-tune and deploy due to massive parameter counts (7B+).
• Specialist systems (e.g., MERaLiON-2-10B-ASR) achieve state-of-the-art (SOTA) performance but require prohibitive computational resources (128 GPUs, ~$18,862 training cost), making them inaccessible to most academic groups and small enterprises.

The core challenge is to develop a compact, cost-effective, and highly accurate multilingual ASR system that handles Singapore's linguistic diversity without relying on proprietary data or massive compute clusters.

2. Methodology

The authors propose Polyglot-Lion, a family of compact models (0.6B and 1.7B parameters) derived from fine-tuning the Qwen3-ASR base models. The methodology relies on two critical design choices:

A. Balanced Multilingual Sampling (Two-Stage Upsampling)

To address the severe data imbalance in public corpora (where English/Mandarin dominate and Tamil/Malay are under-represented), the authors employ a deterministic, two-stage upsampling strategy:

1. Intra-language Balancing: Within each language group, datasets are upsampled to match the size of the largest dataset in that group.
2. Inter-language Balancing: The balanced datasets of all four languages are further upsampled to ensure exact parity (25% contribution per language) in the final training batch.

• Result: This eliminates the need for temperature-based hyperparameter tuning and guarantees that low-resource languages receive equal gradient updates, preventing the model from overfitting to high-resource languages.

B. Language-Agnostic Decoding

Unlike standard multilingual ASR systems (e.g., Whisper) that require explicit language tags as input tokens, Polyglot-Lion is trained and inferred without language conditioning.

• The model must implicitly identify the spoken language based solely on acoustic and linguistic patterns.
• This design is crucial for Singapore, where code-switching is common and pre-labeling audio segments by language is often infeasible.

C. Training Setup

• Base Models: Qwen3-ASR-0.6B and Qwen3-ASR-1.7B.
• Data: Exclusively public corpora (Librispeech, NSC, AISHELL, Common Voice, Fleurs, SLR, Mesolitica) totaling ~969 hours.
• Hardware: Fine-tuned on a single NVIDIA RTX PRO 6000 GPU for 48 hours.

3. Key Contributions

1. Balanced Fine-Tuning Recipe: A simple, hyperparameter-free upsampling strategy that significantly boosts accuracy on low-resource languages (Tamil, Malay) without degrading performance on high-resource ones.
2. Implicit Language Identification: Demonstrates that a moderate-scale model can robustly handle code-switching and identify languages implicitly, removing the dependency on explicit language tags.
3. Comprehensive Benchmarking: Evaluation across 12 standard datasets covering all four official languages, comparing against 8 baselines ranging from general models to large specialist systems.
4. Cost-Efficiency Analysis: A quantitative demonstration that near-SOTA performance can be achieved at a fraction of the computational cost of larger systems.

4. Results

The Polyglot-Lion-1.7B model was evaluated against baselines including Whisper-v3-turbo, SeaLLMs-Audio-7B, and the specialist MERaLiON-2-10B-ASR.

• Accuracy:

  • Polyglot-Lion-1.7B achieved an average error rate of 14.85% across 12 benchmarks.
  • This is nearly identical to MERaLiON-2-10B-ASR (14.32%), a model 6× larger (10B parameters).
  • It significantly outperformed the unmodified Qwen3-ASR-1.7B base (53.76% avg error), proving the efficacy of the fine-tuning recipe.
  • Language-specific gains:
    • Tamil: Reduced WER from >120% (base model) to 39.19% on Common Voice.
    • Malay: Achieved the best result on the Mesolitica benchmark (21.51% WER), outperforming MERaLiON.
    • Mandarin: Achieved the lowest Character Error Rate (CER) across all benchmarks, even beating the 10B specialist model.

• Inference Speed:

  • Polyglot-Lion-1.7B processes audio in 0.10 seconds/sample.
  • This is approximately 20× faster than MERaLiON-2-10B-ASR (2.02 s/sample) and 3× faster than Whisper.

• Training Cost:

  • Polyglot-Lion: ~$81 (1 GPU, 48 hours).
  • MERaLiON-2-10B-ASR: ~$18,862 (128 GPUs, 48 hours).
  • Efficiency: Polyglot-Lion achieves comparable accuracy at 233× lower cost.

5. Significance

The paper demonstrates that data balancing and efficient fine-tuning are more critical factors for multilingual ASR performance than raw model scale or massive compute resources.

• Democratization of AI: By reducing the training cost from ~$19k to ~$80 and enabling training on a single consumer-grade GPU, the authors make high-quality, specialized multilingual ASR accessible to academic researchers and small enterprises.
• Practical Deployment: The language-agnostic design and high inference speed make Polyglot-Lion ideal for real-world Singaporean applications (e.g., customer service, broadcast monitoring) where code-switching is the norm and latency is critical.
• Future Direction: The work suggests that for low-resource languages, targeted data balancing and implicit learning strategies can close the gap with massive specialist models, shifting the focus from "bigger models" to "smarter training."

Limitations: The model still lags slightly behind MERaLiON on Singapore-specific English (Singlish) and Tamil, likely due to the lack of proprietary, domain-specific data in the public training set. Future work aims to incorporate local speech data and cross-lingual text transfer to further close these gaps.