💬 NLP

Closing the Quality Gap in Low-Resource Text-to-Speech: LoRA Fine-Tuning of VoxCPM2 for Khmer and Korean

This paper demonstrates that applying a single Low-Rank Adaptation (LoRA) adapter to the VoxCPM2 model significantly improves the text-to-speech quality for the low-resource Khmer language while maintaining performance for Korean, highlighting that such adaptation is most effective for languages where the base model initially underperforms.

Original authors: Phannet Pov, Sovandara Chhoun, Hyun Woo Park, Wan-Sup Cho, Saksonita Khoeurn

Published 2026-06-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Phannet Pov, Sovandara Chhoun, Hyun Woo Park, Wan-Sup Cho, Saksonita Khoeurn

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a world-class chef who can cook incredible meals for people who speak English, Mandarin, or Spanish. This chef has tasted millions of dishes and knows exactly how to season them. However, if you ask this chef to cook a traditional meal from Cambodia (Khmer) or Korea, they might get the ingredients right, but the flavor is "off." The spices are slightly wrong, the rhythm of the cooking is awkward, and it doesn't quite taste like a home-cooked meal.

This paper is about giving that chef a tiny, specialized "recipe card" to fix their cooking for these specific languages without forcing them to go back to culinary school and relearn everything from scratch.

Here is the breakdown of what the researchers did, using simple analogies:

The Problem: The "Quality Gap"

The researchers used a massive AI model called VoxCPM2. Think of this model as a super-smart robot voice that has listened to thousands of hours of speech.

For big languages (like English): The robot sounds almost human.
For smaller languages (like Khmer): The robot sounds robotic. It mispronounces words and gets the musical "flow" of the sentence wrong.
The Goal: They wanted to fix the robot's voice for Khmer (a language with no spaces between words and very little data available) and Korean (a language the robot already knows quite well).

The Solution: The "LoRA Adapter" (The Recipe Card)

Usually, to fix a robot's voice for a new language, you have to retrain the whole brain. That's like telling the chef to forget everything they know and start over. It takes forever and costs a fortune.

Instead, the researchers used a technique called LoRA (Low-Rank Adaptation).

The Analogy: Imagine the robot's brain is a giant, frozen library of books. You can't change the books. But, you can stick a small, sticky note (the adapter) on the page. This sticky note tells the robot, "When you see a Khmer word, do this specific tweak."
The Magic: They trained just one sticky note to work for both Khmer and Korean at the same time. This note only changed about 0.2% to 3% of the robot's total brain power. It was a tiny, cheap, and fast fix.

The Experiment: Two Different Results

They tested this "sticky note" on two languages to see if it worked.

1. The Khmer Result (The Big Win)

Before: The robot's Khmer voice was decent but flawed (Score: 3.85 out of 5). It sounded a bit stiff.
After: With the sticky note, the voice became much more natural (Score: 4.23 out of 5).
The Catch: They tried different sizes of sticky notes (called "ranks").
- A tiny note (Rank 8) helped a little.
- A medium note (Rank 64) was the perfect size. It fixed the rhythm and intonation beautifully.
- A huge note (Rank 128) actually made it worse, even though the computer's internal math said it was "better."
Lesson: For a language the robot didn't know well, a medium-sized fix was perfect.

2. The Korean Result (The "Don't Touch It" Warning)

Before: The robot already spoke Korean pretty well (Score: 3.65).
After: The sticky note didn't help. In fact, if they used a big sticky note (Rank 64), the robot actually got worse. It started sounding unnatural.
Lesson: If the robot already knows the language well, adding a "fix" just confuses it. You don't need to teach a chef who already knows how to make Kimchi how to make Kimchi again; you might just ruin the recipe.

The Surprising Discovery: "The Computer Lies"

The researchers found something weird.

The computer's internal score (called "Loss") said the biggest sticky note (Rank 128) was the best because the math errors were lowest.
But when real humans listened to the voices, the medium sticky note (Rank 64) was the winner for Khmer.
The Takeaway: Just because the computer's math says "more is better" doesn't mean the human ear agrees. Sometimes, a smaller, simpler fix sounds more natural.

Summary of Findings

One size does not fit all: A single small "adapter" can fix a low-resource language (Khmer) effectively, but it might break a language the model already knows (Korean).
Less is often more: You don't need to change the whole brain. Changing a tiny fraction (less than 3%) is enough to make a huge difference.
Listen to humans, not just math: The best size for the fix was found by listening tests, not by looking at the computer's error charts.
Target the weak spots: This method is most useful for languages where the AI is currently struggling. If the AI is already good, don't mess with it.

In short, the paper shows that you can give a giant AI a tiny, cheap "cheat sheet" to speak a difficult language much better, but you have to be careful not to give it a cheat sheet for a language it already speaks fluently.

Technical Summary: Closing the Quality Gap in Low-Resource Text-to-Speech via LoRA Fine-Tuning of VoxCPM2

Problem Statement
While large pretrained Text-to-Speech (TTS) models achieve near-human quality for well-resourced languages (e.g., English, Mandarin), they exhibit a significant quality gap for low-resource languages. This gap is characterized by mispronunciations, awkward stress and intonation, and a noticeable synthetic quality. The issue is particularly acute for languages like Khmer, which are underrepresented in pretraining data and possess unique orthographic features (e.g., lack of word spacing). Conventional full fine-tuning is computationally expensive, requires separate checkpoints for each language, and risks catastrophic forgetting. Parameter-Efficient Fine-Tuning (PEFT), specifically Low-Rank Adaptation (LoRA), offers a potential solution, but its efficacy in closing the quality gap for low-resource TTS and its ability to serve multiple typologically distinct languages via a single shared adapter remain open questions.

Methodology
The authors investigate these questions using VoxCPM2, a 2.4B-parameter, tokenizer-free TTS model that combines a MiniCPM-4 language model backbone with a flow-matching diffusion decoder.

Corpus Construction: A shared, language-tagged corpus was created comprising approximately 26 hours of audio from Khmer (3.96 hours) and Korean (22.21 hours). The Khmer data was sourced from the Institute of Digital Research & Innovation (Cambodia), while Korean data came from the KSS corpus and Common Voice/FLEURS. To address data scarcity, Khmer training clips were upsampled to constitute 40% of the training mix.
Model Adaptation: Instead of full fine-tuning, a single shared LoRA adapter was attached to the frozen VoxCPM2 backbone. This adapter was applied to the query, key, value, and output projections of the attention mechanisms in both the MiniCPM-4 language model (base and residual layers) and the flow-matching diffusion decoder.
Training Configuration: The adapter was zero-initialized ( $\Delta W = 0$ ), ensuring training began exactly at the zero-shot baseline. The study tested five rank configurations ( $r \in \{8, 16, 32, 64, 128\}$ ), representing 0.19% to 3.03% of the total parameters. Training utilized the AdamW optimizer with mixed precision on a single NVIDIA H200 GPU.
Evaluation: Performance was measured via:
1. Automatic Metrics: Validation flow-matching loss and stop-token loss.
2. Human Evaluation: Native-speaker Mean Opinion Scores (MOS) on a 5-point scale across three axes: naturalness, prosody, and pronunciation. Statistical significance was assessed using paired Wilcoxon signed-rank tests.

Key Results
The study yielded divergent results for the two languages, highlighting the dependency of adaptation efficacy on the base model's initial proficiency:

Khmer (Low-Resource): The adapter produced a highly significant improvement in quality. The overall MOS increased from 3.85 (zero-shot) to 4.23 at rank 64 ( $p < 0.001$ ). The most substantial gain was observed in prosody (rising from 3.76 to 4.36), indicating the base model's primary weakness was rhythm and intonation. Notably, even the smallest adapter (rank 8) recovered most of the gain, achieving a MOS of 3.91.
Korean (Well-Resourced): The base model already handled Korean well, and the adapter provided no significant overall improvement. In fact, higher ranks (specifically rank 64) significantly degraded quality (MOS dropped to 3.48, $p = 0.02$ ), primarily due to a loss in naturalness. This suggests that for languages the model already knows well, additional capacity leads to overfitting on the small fine-tuning dataset.
Loss vs. Perception Discrepancy: A critical finding is the misalignment between automatic loss and human perception. The validation loss was lowest at rank 128 (0.7094), yet the highest MOS for Khmer occurred at rank 64. The loss continued to decrease at rank 128, but perceptual quality dropped, suggesting that validation loss overstates the value of extra capacity.

Key Contributions

Shared Multilingual Adapter: The authors present the first parameter-efficient adaptation of a foundation TTS model for Khmer. They demonstrate that a single, small LoRA adapter (0.19–3.03% of parameters) can effectively serve two typologically distinct languages (Khmer and Korean) simultaneously without requiring separate models.
Targeted Adaptation Efficacy: The study empirically validates that adaptation yields significant gains only where the base model is genuinely weak. The "useful" adapter size scales with the distance between the language and the model's existing knowledge; large gaps (Khmer) absorb more capacity, while small gaps (Korean) suffer from overfitting.
Rank Selection Insight: The paper challenges the reliance on validation loss for hyperparameter selection in TTS. It demonstrates that the optimal rank for perceptual quality (rank 64 for Khmer) does not necessarily align with the lowest validation loss (rank 128), advocating for listening tests as the primary metric for rank selection.
Diagnostic Utility: The authors propose that the magnitude of improvement provided by a zero-initialized adapter serves as a probe for the base model's existing knowledge of a language.

Significance and Claims
The paper claims that small LoRA adapters are a highly effective, resource-efficient mechanism for closing the quality gap in low-resource TTS, provided the base model lacks sufficient prior knowledge of the target language. The study emphasizes that a "one-size-fits-all" global rank is inappropriate; instead, rank selection must be language-specific and guided by perceptual evaluation rather than automatic loss.

The authors modestly note limitations, including the lack of a full fine-tuning upper bound, the potential confounding factors between language coverage and data source quality (private vs. public corpora), and the relatively small rater panels used for MOS evaluation. They conclude that future work should focus on establishing full fine-tuning baselines and expanding evaluation to include more low-resource languages and objective metrics.