Benchmarking Large Language Models for Grapheme-to-Phoneme Conversion: A Japanese Case Study
This paper benchmarks over 30 large language models for Japanese grapheme-to-phoneme conversion, demonstrating that specialized models using a parse mode with rule-based post-processing significantly outperform conventional tools and end-to-end text-to-speech systems in pronunciation accuracy.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are trying to teach a robot how to speak Japanese. The robot can read the written characters (like Kanji, which look like complex pictures), but it doesn't automatically know how to say them out loud. This is the job of Grapheme-to-Phoneme (G2P) conversion: turning written text into the sounds (phonemes) needed to speak.
In Japanese, this is a nightmare for computers because:
- No Spaces: Words run together without spaces, so the computer has to guess where one word ends and the next begins.
- Many Meanings: One character can have many different sounds depending on the context (like the English word "read" sounding different in "I read a book" vs. "I will read a book").
- Tricky Rules: Small grammar particles change their sound based on what comes next, and numbers combined with counters (like "one cup" vs. "two cups") have completely irregular sounds.
For years, we used specialized, rule-heavy software (like a strict dictionary) to solve this. But recently, Large Language Models (LLMs)—the same AI brains that write essays and chat with us—have become powerful enough to try this job too.
This paper is a big race to see which AI is the best at reading Japanese text aloud.
The Two Strategies: "The Translator" vs. "The Magician"
The researchers tested two ways to ask the AI to do this:
The "Translator" Mode (Parse Mode):
- How it works: You ask the AI to first break the sentence down into individual words (like a dictionary lookup) and write down the sound for each word in a structured list. Then, a simple set of computer rules (like a spell-checker) fixes the final sounds (e.g., changing "ha" to "wa" for specific grammar particles).
- The Analogy: This is like hiring a translator to write a word-for-word glossary, and then having a human editor quickly fix the grammar at the end. The AI doesn't have to worry about the complex rules; it just has to identify the words.
The "Magician" Mode (Direct Mode):
- How it works: You tell the AI, "Here is the sentence, just give me the final sound in one go." The AI has to do everything at once: find the words, guess the sounds, and apply all the tricky grammar rules simultaneously.
- The Analogy: This is like asking a magician to pull a rabbit, a dove, and a dove out of a hat in a single, seamless motion. It's elegant, but if the AI misses one tiny rule, the whole trick fails.
The Race Results
The researchers tested over 30 different AI models (from small, cheap ones to massive, expensive ones) on 3,000 sentences. Here is what they found:
Bigger is Better: Just like a bigger brain can remember more facts, bigger AI models made far fewer mistakes. The smallest models were confused and often guessed random words, while the largest ones were incredibly accurate.
Special Training Matters: AI models that were specifically "taught" more Japanese during their training (called "Japanese-specialized") were much better than generic models, even if the generic models were huge.
The Winner: The "Translator" (Parse) mode was the clear winner for almost everyone. By letting the AI focus only on identifying words and letting a simple rule-checker handle the complex grammar, the errors dropped significantly.
- The Best Result: The top AI (Claude Opus 4.6) made mistakes in less than 0.52% of the characters.
- The Old Guard: The best traditional software (OpenJTalk) made mistakes in 1.03% of the characters.
- The Takeaway: The new AI is roughly twice as accurate as the old standard tools.
When the "Magician" Wins: Interestingly, for a few of the absolute smartest AI models, the "Magician" (Direct) mode worked slightly better. These super-smart models were so good at following complex instructions that they didn't need the help of the rule-checker. However, for most models, trying to do everything at once led to confusion.
Does it actually sound good?
The researchers didn't just check if the AI got the letters right; they checked if it made a robot voice sound natural.
They took the sounds generated by the AI and fed them into a Text-to-Speech (TTS) system. They compared this to "End-to-End" systems (where the AI tries to turn text directly into voice without a middle step).
- The Result: The AI-assisted method (using the "Translator" mode) produced much clearer pronunciation (fewer wrong words) than the End-to-End systems.
- The Catch: Despite being more accurate, it sounded just as natural and human-like as the End-to-End systems.
The Bottom Line
This paper proves that if you want a robot to speak Japanese clearly, especially for tricky names or words not found in standard dictionaries, using a modern Large Language Model is the best approach.
However, you shouldn't just ask the AI to "do it all." The secret sauce is to ask the AI to break the sentence into words first (Parse Mode), and then let a simple computer program fix the final sounds. This combination creates a voice that is both incredibly accurate and surprisingly human.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.