VietNormalizer: An Open-Source, Dependency-Free Python Library for Vietnamese Text Normalization in TTS and NLP Applications

This paper introduces VietNormalizer, a lightweight, zero-dependency Python library that provides a comprehensive, rule-based pipeline for normalizing diverse Vietnamese text elements—such as numbers, dates, currency, and foreign terms—into pronounceable forms for TTS and NLP applications.

The Gist

Imagine you are trying to teach a robot to speak Vietnamese. You give it a text message like: "Hôm nay lúc 9:30, giá container là 1.500.000đ, gặp NASA ở Singapore."

If you ask the robot to read this out loud exactly as written, it would sound like a glitchy robot: "Hôm nay lúc chín colon ba mươi, giá container là một chấm năm chấm không... NASA ở Singapore." It would stumble over the numbers, the time, the currency, and the English words.

VietNormalizer is the "translator" that fixes this mess before the robot tries to speak. It's a free, lightweight tool that turns messy, written text into clean, spoken Vietnamese.

Here is a simple breakdown of what the paper is about, using some everyday analogies:

1. The Problem: The "Raw Ingredient" vs. The "Cooked Meal"

Think of raw text (like social media posts or news articles) as a pile of raw ingredients. It has numbers, dates, abbreviations, and foreign words mixed in.

• The Issue: A Text-to-Speech (TTS) engine is like a chef who only knows how to cook with pre-chopped, pre-measured ingredients. If you hand the chef a whole potato (a number like "1,500,000") or a foreign spice name ("container"), the chef gets confused and makes a mess.
• The Gap: Before this tool, people had to either:
  • Use heavy, expensive machinery (massive AI models) just to chop the potatoes.
  • Use a tiny, broken knife that could only chop potatoes but not onions (tools that only handle numbers but ignore dates or money).
  • Or, they had no tool at all.

2. The Solution: The "Magic Kitchen Assistant"

VietNormalizer is like a super-efficient, zero-maintenance kitchen assistant.

• No Heavy Machinery: It doesn't need a giant supercomputer (GPU) or a massive database of AI models to work. It runs on a simple laptop or even a tiny phone. It's "dependency-free," meaning you don't need to install a bunch of other heavy software to use it.
• The Rulebook: Instead of "guessing" like a human might, it follows a strict, pre-written recipe book (rules).
  • Rule: If you see "9:30", say "chín giờ ba mươi phút" (nine o'clock thirty minutes).
  • Rule: If you see "1.500.000đ", say "một triệu năm trăm nghìn đồng" (one million five hundred thousand dong).
  • Rule: If you see "NASA", say "na-sa" (how it sounds in Vietnamese).

3. What Does It Actually Do? (The 7 Superpowers)

The paper lists seven specific things this tool fixes, which we can think of as its "superpowers":

1. Number Wizard: Turns "123" into "one hundred twenty-three" (with all the tricky Vietnamese grammar rules).
2. Time Traveler: Turns "14:30" into "two thirty PM" style Vietnamese.
3. Money Counter: Handles currency (VND and USD) so it sounds natural.
4. Percentage Pro: Turns "50%" into "fifty percent."
5. Acronym Decoder: Turns "GDP" into "tổng sản phẩm quốc nội" (Gross Domestic Product).
6. Foreign Word Translator: Takes English words like "container" and turns them into their Vietnamese sound-alike "công-te-no."
7. Cleanup Crew: Removes emojis and weird symbols that robots hate.

4. Why Is This Better Than the Old Ways?

The authors compare their tool to previous attempts using a "Speed vs. Weight" analogy:

• Old AI Tools: Imagine a Ferrari that needs a team of mechanics and a full tank of premium fuel (huge computer power) just to drive to the grocery store. It's fast on the highway (very accurate on perfect data) but breaks down if the road is bumpy (social media slang) or if you don't have the fuel (computing power).
• VietNormalizer: This is a bicycle. It's light, you can ride it anywhere, it doesn't need fuel, and it gets the job done instantly. It's built for speed and reliability in real-world situations.

5. Why Should We Care?

• For Developers: It's free, easy to install (just one line of code), and works instantly.
• For the World: The paper argues that this "rule-based" approach (using a recipe book instead of a guessing AI) is the best way to help other languages that don't have enough data to train big AI models. If you have a language with few speakers, you can't feed a computer millions of examples. But you can write a rulebook. VietNormalizer proves this works for Vietnamese, and the blueprint can be used for many other languages.

The Bottom Line

VietNormalizer is a free, open-source tool that acts as a bridge between messy written text and clear spoken Vietnamese. It strips away the need for heavy, expensive AI, allowing anyone to build robots that can speak Vietnamese naturally, quickly, and without needing a supercomputer.

Technical Summary

Here is a detailed technical summary of the paper "VietNormalizer: An Open-Source, Dependency-Free Python Library for Vietnamese Text Normalization in TTS and NLP Applications."

1. Problem Statement

Text Normalization (TN) is the critical preprocessing step in Text-to-Speech (TTS) and Natural Language Processing (NLP) pipelines where written text containing Non-Standard Words (NSWs) is converted into a canonical, fully pronounceable spoken form.

For Vietnamese, this task presents unique challenges:

• High Density of NSWs: Real-world text (social media, news, commerce) frequently mixes Arabic numerals, dates, times, currency, percentages, acronyms, and foreign loanwords within a single sentence.
• Linguistic Complexity: Vietnamese is a tonal language with six lexical tones. Incorrect tone transcription during normalization leads to immediate mispronunciation in TTS. Additionally, number verbalization follows irregular grammatical patterns (e.g., mot vs. mot muoi) distinct from other Southeast Asian languages.
• Gap in Existing Tools: Current solutions suffer from significant limitations:
  • Neural Approaches: Tools like Trang et al. (2022) and ViSoLex offer high accuracy but require heavy dependencies (BERT, PyTorch/TensorFlow), large model sizes (GBs), and high inference latency, making them unsuitable for lightweight or real-time deployment.
  • Toolkit-Embedded Tools: Libraries like underthesea focus on Unicode standardization but lack specific NSW expansion (numbers, currency, acronyms) and carry unnecessary overhead for users needing only TN.
  • Academic Prototypes: Early works (e.g., Tuan et al., 2012) were not distributed as standalone, installable libraries.

2. Methodology

VietNormalizer addresses these gaps through a pure Python 3.8+ rule-based pipeline designed for zero external dependencies.

Architecture

The system is built around two primary classes: VietnameseTextProcessor (core rules) and VietnameseNormalizer (API composition). The normalization pipeline executes transformations in a strict, priority-based order to prevent pattern conflicts:

1. Unicode Normalization: NFC normalization and removal of emojis/non-printable characters.
2. Date & Time: Converts patterns (e.g., DD/MM/YYYY, HH:MM) to spoken Vietnamese (e.g., chin gio ba muoi phut).
3. Currency: Verbalizes VND and USD amounts (e.g., mot trieu nam tram nghin dong).
4. Percentages: Converts "X%" to "X phan tram".
5. General Numbers: Uses a recursive decomposition algorithm to handle integers and decimals, accounting for irregularities (e.g., linh mot vs. mot muoi) and large number groupings (nghin, trieu, ty).
6. Dictionary Replacement: A single-pass regex substitution for acronyms and loanwords.

Key Technical Features

• Zero-Dependency: Requires no neural networks, external APIs, or heavy libraries.
• Performance Optimization: All regular expressions are pre-compiled at initialization. Dictionary keys are combined into a single alternation regex, ensuring $O(n)$ processing time relative to input length, regardless of dictionary size.
• Extensibility: Users can provide custom CSV files for acronyms and loanwords, which are loaded at runtime or via initialization.
• Transliteration: Includes a mechanism to convert foreign loanwords (English, French, etc.) into Vietnamese phonetic approximations (e.g., container → cong-te-no).

3. Key Contributions

1. Comprehensive Rule-Based Pipeline: The first standalone library covering seven major NSW categories (Numbers, Dates, Times, Currency, Percentages, Acronyms, Loanwords) specifically for Vietnamese.
2. Production-Ready Design: Achieves high-throughput batch processing (tens of thousands of utterances per minute on a single CPU core) with minimal memory overhead, making it suitable for large-scale TTS data preparation (e.g., the VietSuperSpeech pipeline).
3. User-Extensible Dictionary System: A CSV-based mechanism allowing users to customize acronym expansion and loanword transliteration without modifying the core code.
4. Open-Source Release: Available on PyPI (pip install vietnormalizer) and GitHub under the MIT license, ensuring immediate accessibility.
5. Generalizability Framework: The paper proposes a tiered framework for adapting this rule-based architecture to other low-resource, tonal, and agglutinative languages (e.g., Thai, Lao, Mandarin, Indonesian).

4. Results and Performance

• Efficiency: By avoiding neural inference, the library eliminates the latency (tens to hundreds of milliseconds) associated with BERT-based taggers, reducing processing time to microseconds per utterance.
• Reliability: The deterministic, rule-based approach ensures fully auditable outputs, avoiding the error propagation risks inherent in neural sequence tagging (where a mis-tagged NSW ruins the normalization).
• Coverage: Unlike underthesea (which only handles Unicode) or ViSoLex (focused on social media slang), VietNormalizer covers the full spectrum of NSWs required for professional TTS synthesis.

5. Significance and Future Work

Significance:
VietNormalizer fills a critical void in the Vietnamese NLP ecosystem by providing a lightweight, installable, and high-performance tool that bridges the gap between raw text and TTS-ready phonemes. It demonstrates that for low-resource languages lacking large labeled datasets for neural training, rule-based paradigms remain the most practical and deployable solution. The paper argues that this approach is not just a stopgap but a robust baseline for the majority of the world's 7,000 languages.

Limitations & Future Work:

• Contextual Ambiguity: The current system relies on pattern priority (e.g., matching dates before fractions) and cannot fully resolve ambiguities requiring deep syntactic parsing (e.g., distinguishing "2/9" as a date vs. a fraction without context).
• Named Entities: Proper nouns (names, locations) are not yet fully handled without external NER integration.
• Code-Switching: Future versions aim to integrate lightweight token-level language identification to better handle mixed-language utterances.
• Inverse Text Normalization (ITN): Converting spoken text back to written form is planned for a future release.

In conclusion, VietNormalizer represents a significant step forward in democratizing high-quality TTS for Vietnamese, offering a replicable blueprint for text normalization in other resource-constrained linguistic environments.