🤖 AI

MERaLiON-GR: Speech Gender Recognition Model for English and SEA Languages

The paper introduces MERaLiON-GR, a speech gender recognition model that leverages LoRA fine-tuning of the MERaLiON-SpeechEncoder-2 and a multi-scale ECAPA-TDNN classifier to achieve state-of-the-art performance in identifying male and female speakers across English and multiple Southeast Asian languages.

Original authors: Qiongqiong Wang, Ai Ti Aw, Nancy F. Chen, Ying Lay Chiu, Yang Ding, Yingxu He, Ridong Jiang, Zhuohan Liu, Yanfeng Lu, Yi Ma, Muhammad Huzaifah, Nabilah Binte Md Johan, Nattadaporn Lertcheva, Pham Minh

Published 2026-08-06

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Qiongqiong Wang, Ai Ti Aw, Nancy F. Chen, Ying Lay Chiu, Yang Ding, Yingxu He, Ridong Jiang, Zhuohan Liu, Yanfeng Lu, Yi Ma, Muhammad Huzaifah, Nabilah Binte Md Johan, Nattadaporn Lertcheva, Pham Minh Duc, Sailor Hardik Bhupendra, Siti Umairah Binte Mohammad Salleh, Shuo Sun, Tarun Kumar Vangani, Jeremy H. M. Wong, Jinyang Wu, Longyin Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are at a bustling, noisy party where people are speaking different languages, laughing, shouting, and whispering all at once. If you close your eyes, you can often tell who is a man and who is a woman just by the sound of their voice. Your brain does this automatically, mixing together clues like pitch, texture, and rhythm. But teaching a computer to do the same thing is like trying to teach a robot to recognize a friend's voice in a hurricane. This field of science is called speech gender recognition. It's a branch of artificial intelligence that tries to figure out if a speaker is male or female just by listening to their voice, without seeing them.

To understand how computers learn this, think of two different tools. The first is like a specialized detective. This tool has been trained specifically to listen for voice clues. It ignores what the person is saying and focuses entirely on how they say it. The second tool is like a general knowledge encyclopedia. This is a massive AI that has read almost everything and can answer questions about the world, but it hasn't been specifically trained to be a voice detective. For a long time, scientists wondered: "Do we need a specialized detective, or is the encyclopedia smart enough to figure it out on its own?" This is the big question the researchers in this paper set out to answer. They wanted to build a voice detective that works not just in English, but also in the many languages spoken across Southeast Asia, and they wanted to see if it could beat the best tools currently available.

The New Voice Detective: MERaLiON-GR

The team behind this paper, known as the MERaLiON Team, has built a new voice detective called MERaLiON-GR. Think of this model as a super-smart robot ear that has been trained on a massive library of voices from English and eight different Southeast Asian languages, including Chinese, Malay, Tamil, Thai, Vietnamese, Indonesian, and Khmer.

How it works (The Recipe)
Imagine the robot's brain is built in three layers:

The Big Brain (The Backbone): At the core is a giant, pre-trained brain called MERaLiON-SpeechEncoder-2. This brain has already listened to millions of hours of speech and knows how sounds work. However, the team didn't want to retrain the whole brain from scratch because that would be like trying to rewrite an entire encyclopedia just to add a new chapter on "voice gender." That takes too much energy and time.
The Smart Adapter (LoRA): Instead of rewriting the whole brain, they used a clever trick called LoRA (Low-Rank Adaptation). Imagine the big brain is a giant, heavy library. LoRA is like adding a small, lightweight set of sticky notes to the shelves. These notes tell the library how to rearrange its existing knowledge specifically for the task of spotting male vs. female voices. It's efficient, fast, and doesn't mess up the library's other knowledge.
The Final Judge (ECAPA-TDNN): The brain then passes its findings to a specialized judge called an ECAPA-TDNN. This judge looks at the clues from different layers of the brain, combines them, and makes the final decision: "Male" or "Female."

The Big Test
The team put their new detective to the test against two other contenders:

Vox-Profile: The current champion of voice gender recognition.
MERaLiON-v2: A general-purpose "encyclopedia" AI (an Audio-LLM) that can talk and listen but isn't a specialized voice detective.

They tested everyone on a huge variety of voices: clear studio recordings, messy street noise, short 2-second clips, and long conversations. The languages ranged from standard English to the unique mix of Singlish and various Southeast Asian dialects.

The Results: Specialized Wins
The results were clear. The specialized detective, MERaLiON-GR, crushed the competition in almost every category.

Beating the Champion: On 12 out of 15 different test sets, MERaLiON-GR was more accurate than the current champion, Vox-Profile. In some cases, like with Tamil and Thai elderly voices, it got a perfect score of 100.00%.
The "Wild" Test: The real challenge was "in-the-wild" recordings—short, messy clips from real life (10 to 30 seconds long). Here, MERaLiON-GR was still the winner, beating Vox-Profile by up to 4.32 percentage points. This is a big deal because real-world audio is full of background noise and short snippets that confuse other models.
The Encyclopedia's Struggle: The general-purpose AI (the "encyclopedia") did okay, but it consistently lost to the specialized detective. Without specific training, it struggled to get the fine details right. For example, on the English FLEURS test, the encyclopedia only got 49.61% right, while the specialized detective got 100.00%.

The Magic Trick: Helping the Encyclopedia
Here is the most interesting part. The team discovered that even though the general-purpose AI (the encyclopedia) isn't a great voice detective on its own, it becomes much smarter if you just tell it the answer first.
When they took the gender guess from their specialized detective (MERaLiON-GR) and gave it to the encyclopedia as a "hint" (metadata), the encyclopedia's performance skyrocketed.

On Vietnamese Common Voice, the encyclopedia jumped from 36.08% to 96.08% accuracy just by being told the gender first.
On English FLEURS, it jumped from 49.61% to 97.31%.

This suggests that while general AI models are powerful, they still need a specialized "assistant" to handle specific tasks like identifying gender. Once they have that clue, they can use their massive knowledge to understand the rest of the conversation much better.

Why This Matters
The paper concludes that for tasks like recognizing gender in a mix of languages and noisy environments, you really do need a dedicated, specialized tool. You can't just rely on a general AI to figure it out. The MERaLiON-GR model shows that by using a smart, efficient method (LoRA) to adapt a large brain to a specific job, we can build systems that understand the diverse voices of Southeast Asia better than ever before. It's a reminder that sometimes, the best way to solve a specific problem is to build a specialist, not just rely on a generalist.

Technical Summary: MERaLiON-GR

Problem Statement

Speech gender recognition (GR) is a fundamental paralinguistic task essential for speaker diarization, profiling, and forensic applications. While human listeners can infer gender from brief utterances, automatic systems struggle to generalize across diverse recording conditions, languages, accents, age groups, and channel characteristics. Existing approaches face two primary challenges:

Generalization: Traditional methods relying on handcrafted features (e.g., F0, formants) or even modern deep learning models often fail to maintain robustness across the wide linguistic diversity of Southeast Asia (SEA) and English.
Efficiency vs. Performance: Fully fine-tuning large pre-trained speech models (e.g., 24+ layers) for specific downstream tasks is computationally expensive and risks catastrophic forgetting of the pre-trained representations.

Methodology

The authors propose MERaLiON-GR, a binary classification system (female/male) designed for English and Southeast Asian languages. The architecture integrates a large pre-trained backbone with a parameter-efficient adaptation strategy and a specialized downstream network.

1. Backbone: MERaLiON-SpeechEncoder-2 with LoRA

The core of the system is MERaLiON-SpeechEncoder-2, a large Conformer-based transformer pre-trained on a multilingual corpus heavily featuring Singaporean English and other SEA languages.

Parameter-Efficient Fine-Tuning (PEFT): Instead of updating all 24 transformer layers, the authors employ Low-Rank Adaptation (LoRA). LoRA adapters are inserted into the attention projection layers, parameterizing weight updates via low-rank factors (rank and scaling factor set to 16). This reduces computational cost and overfitting risks.
Normalization and Regularization: The model utilizes rsLoRA normalization and a LoRA dropout rate of 0.05.
Layer Aggregation: All 25 hidden-state outputs (input embeddings + 24 layers) are retained. A learned soft-attention mechanism aggregates these per-layer mean representations into a weighted combination ( $H = \sum a_l h_l$ ), capturing complementary phonetic and speaker-level cues.

2. Downstream Network: ECAPA-TDNN

The aggregated representation is processed by an ECAPA-TDNN network adapted for gender recognition:

Architecture: The input is projected to 512 dimensions and passed through three SE-Res2Net blocks with dilations $\{1, 2, 3\}$ and an SE reduction factor of 8.
Normalization: All BatchNorm layers are replaced with GroupNorm layers to match the original ECAPA-TDNN structure.
Pooling and Classification: Attention pooling aggregates temporal frames into a fixed-length embedding, which is projected to 256 dimensions. A linear classification head with RMSNorm and GELU activation maps this to two class logits.

3. Training Strategy

Datasets: Training utilizes six partitions: VoxCeleb1 (gender-labeled) and five partitions of the IMDA Singapore speech corpus (covering prompted scripts, spontaneous conversation, code-switching, and stylized speech).
Optimization: The model uses a multi-dataset training strategy with class-balanced weighted cross-entropy loss to address class imbalance.
Hyperparameters: The LoRA adapters and downstream network use distinct learning rates ( $5\times10^{-5}$ and $6\times10^{-4}$ , respectively). Training employs a cosine scheduler with linear warm-up and label smoothing ( $\epsilon=0.1$ ).
Ensembling: To improve robustness without inference overhead, the final model parameters are an average of the four checkpoints with the lowest categorical loss on the development set.

Key Contributions

The paper outlines three primary contributions:

A Parameter-Efficient Multilingual GR Model: A model built on MERaLiON-SpeechEncoder-2 using LoRA and ECAPA-TDNN that outperforms the state-of-the-art Vox-Profile on 12 of 15 public benchmarks across eight languages.
Robust Segment-Level Recognition: Demonstrated capability to perform accurate gender recognition on short, in-the-wild speech segments (10–30 seconds), achieving gains of up to 4.32 percentage points (pp) over Vox-Profile on Singaporean language sets.
Enhanced Audio-LLM Performance: A demonstration that injecting GR estimates as paralinguistic metadata substantially improves the performance of general-purpose Audio-LLMs on gender-related question answering, validating the utility of explicit paralinguistic cues.

Results

Public Benchmark Performance

Evaluated across eight languages (English, Chinese, Malay, Tamil, Thai, Vietnamese, Indonesian, Khmer) on 15 test sets:

Accuracy: MERaLiON-GR achieved higher accuracy than Vox-Profile on 12 of 15 test sets and matched performance on the Khmer FLEURS benchmark.
Perfect Scores: The model attained 100.00% accuracy on English FLEURS, Tamil OpenSLR, and Thai Elderly benchmarks.
Significant Gains: Notable improvements were observed on Tamil EmoTa (+4.71 pp), Vietnamese Common Voice (+3.14 pp), and Indonesian IndoWave (+3.00 pp).
Limitations: Performance decreased on Malay SMALDUSC (-4.40 pp) and Thai SER (-2.09 pp), attributed to domain mismatches in recording conditions and speaking styles compared to the training data.

In-the-Wild Performance

On a manually curated Singapore evaluation set (SG-ECMT) consisting of 10–30 second segments:

MERaLiON-GR consistently outperformed Vox-Profile across all four languages (Singlish, Chinese, Malay, Tamil).
The largest improvement was on Malay (+4.32 pp), followed by Tamil (+4.16 pp), highlighting the model's robustness for fine-grained, segment-level pseudo-labeling.

Audio-LLM Integration

When predicted gender was injected as metadata into the Audio-LLM (GR-AudioLLM):

Accuracy improved significantly across all evaluated datasets compared to the base Audio-LLM.
Gains were particularly dramatic on Vietnamese Common Voice (+60.00 pp), English FLEURS (+47.70 pp), and Malay SMALDUSC (+41.20 pp), confirming that explicit paralinguistic cues act as effective conditioning signals for speech-language understanding.

Significance and Claims

The paper claims that MERaLiON-GR underscores the continued value of dedicated speech models for achieving accurate paralinguistic understanding and strong cross-lingual generalization. The results suggest that:

Specialization Matters: General-purpose Audio-LLMs are less effective for fine-grained gender recognition without task-specific optimization.
Efficiency is Viable: Parameter-efficient fine-tuning (LoRA) on large pre-trained encoders can yield state-of-the-art performance without the computational cost of full fine-tuning.
Metadata Utility: Explicitly providing paralinguistic metadata (like gender) significantly enhances the capabilities of multimodal models in understanding speech context, supporting the integration of specialized encoders into broader AI pipelines.

The work positions MERaLiON-GR as a practical replacement for external GR models, particularly for applications requiring robust performance across diverse Southeast Asian languages and unconstrained recording environments.