💬 NLP

How Tokenization Limits Phonological Knowledge Representation in Language Models and How to Improve Them

This paper demonstrates that standard subword tokenization systematically impairs language models' phonological knowledge representation, and proposes a lightweight IPA-based fine-tuning method that significantly improves phonological performance while largely preserving general reasoning capabilities.

Original authors: Disen Liao, Freda Shi

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Disen Liao, Freda Shi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant student (a Large Language Model, or LLM) how to understand the music of language—how words rhyme, how they are broken into beats (syllables), and how they sound when spoken.

The problem is, this student has never heard a single spoken word. They have only ever read text. And the way we give them text to read is the root of their confusion.

Here is the story of the paper, broken down into simple concepts and analogies.

1. The Problem: The "Lego Brick" Mistake

Imagine you are trying to teach someone how to build a house by showing them a blueprint. But, instead of showing them whole bricks, you hand them a pile of randomly chopped-up Lego pieces.

The Reality: When we feed text to AI models, we don't give them whole words like "musical" or "nation." We chop them up into "subwords" (like mus and ical, or na and tion) based on how often those chunks appear in the training data. This is called Tokenization.
The Issue: These "Lego pieces" (tokens) are cut based on spelling frequency, not sound.
- Rhyme Problem: The words "nation" and "station" rhyme perfectly. But if the AI sees them as na + tion and sta + tion, it might miss the connection because the first chunks (na vs sta) look totally different. It's like trying to match two puzzle pieces where the edges don't quite line up.
- Syllable Problem: The word "musical" has three beats (syllables): mu-si-cal. But the AI might see it as two chunks: mus-ical. It's like trying to count the beats in a song, but the music player is skipping every other beat.

The paper argues that because the AI is forced to read these "chopped-up" pieces, it struggles to learn the natural rhythm and sound of words, even though it's incredibly smart at other things like math or writing essays.

2. The Investigation: The "X-Ray" Test

To prove this, the researchers acted like doctors giving the AI an X-ray. They didn't just ask the AI, "Do these words rhyme?" (which is a performance test). Instead, they looked inside the AI's "brain" (its hidden layers) to see if the concept of rhyming was actually stored there.

The Findings: They found that the AI does know about sounds, but it's buried deep inside.
The Metric (STAD): They invented a ruler called STAD (Syllabification-Tokenization Alignment Distance). Think of it as a "Misalignment Score."
- If the AI's "chopped pieces" line up perfectly with the natural syllables of a word, the score is 0 (Perfect alignment).
- If the pieces are cut in weird places (like cutting "musical" between s and i), the score goes up.
- The Result: The higher the Misalignment Score, the worse the AI gets at understanding sounds. It's like trying to run a race while wearing shoes that are two sizes too big; you can still run, but you'll trip over your own feet.

3. The Culprit: "Foreign Borrowings"

Why do some words get chopped up so badly? The researchers found that words with a lot of history are the troublemakers.

The Analogy: Think of a word like "Musical." It has traveled through French, German, Spanish, and many other languages. In every language, it's spelled slightly differently (musikal, müzikal, musical).
Because the AI has seen so many different spellings of this word in its training data, it gets confused about where to cut it. It creates a "token" that doesn't match the sound. It's like a librarian who has seen the same book in 10 different languages and doesn't know which shelf to put it on.

4. The Solution: The "Sound Tutor"

The researchers asked: Can we fix this without rebuilding the whole AI? (Changing the tokenizer is like rebuilding the library from scratch; it's hard and expensive).

Their Fix: They gave the AI a "Sound Tutor" through a special training method.

The Method: They took the AI's existing knowledge and fed it a new diet of questions. But this time, they added IPA (International Phonetic Alphabet) tags.
- Old way: "Does 'cat' rhyme with 'hat'?"
- New way: "Does cat rhyme with hat? (Hint: They both end in /æt/)."
The Result: The AI learned to pay attention to the sound tags, even though it was still reading text.
- It got much better at rhyming, counting syllables, and translating spelling to sound.
- The Trade-off: It got slightly worse at math and general trivia (about 1% drop), but it kept almost all its general smarts. It's like a student who spends extra time studying music theory; they might be slightly slower at solving math problems for a week, but they can now write a beautiful poem.

Summary: The Big Takeaway

This paper tells us that how we chop up words matters more than we thought.

If you want an AI to understand the music of language, you can't just feed it chopped-up Lego bricks. You either need to:

Cut the bricks better (Design new tokenizers that respect syllables).
Teach the AI to ignore the bad cuts (Use the "Sound Tutor" method to help it focus on the sounds hidden inside the text).

The researchers showed that with a little bit of "sound training," even text-only AI can learn to hear the rhythm of words, opening the door for better poetry generators, language learning apps, and more human-like assistants.

1. Problem Statement

Despite achieving remarkable success in text-based tasks, Large Language Models (LLMs) often struggle with phonological reasoning (e.g., rhyming, syllable counting, and grapheme-to-phoneme conversion). This is puzzling because text-only LMs are never trained on audio data; they must infer sound structures solely from orthography (spelling).

The authors hypothesize that the root cause of these failures lies in tokenization. Standard subword tokenization methods (like Byte Pair Encoding - BPE, or SentencePiece) prioritize statistical frequency over linguistic structure. Consequently:

Coarse Granularity: Tokens may be too large to capture local phonological features (e.g., rhyming endings).
Misalignment: Token boundaries often do not align with natural syllable boundaries, disrupting the model's ability to learn prosodic structures (rhythm and stress).

2. Methodology

The authors employ a three-pronged experimental approach:

A. Experiment 1: Probing Phonological Knowledge (Representation Analysis)

Goal: Determine if LMs encode phonological information in their hidden states despite poor performance on surface tasks.
Method: They use linear probing (training lightweight classifiers/regressors on top of frozen LM hidden states) across different layers.
Tasks:
1. Rhyming Awareness: Binary classification (do two words rhyme?).
2. Grapheme-to-Phoneme (G2P): Regression to predict ARPAbet phoneme indices.
3. Syllable Counting: Regression to predict the number of syllables.
Models Tested: BERT, GPT-2, GPT-Neo, Mistral-7B, Llama-3/3.1-8B.

B. Experiment 2: Quantifying Tokenization Impact

Goal: Isolate the effect of tokenization on phonological performance.
Local Features (Rhyming): They compare standard tokenization against character-level splitting (inserting delimiters like / between characters, e.g., b/o/y). This forces the model to process finer-grained units.
Global Features (Prosody): They introduce a new metric, Syllabification-Tokenization Alignment Distance (STAD).
- Definition: STAD measures the normalized Hamming distance between token split positions and natural syllable boundaries.
- Hypothesis: Lower STAD (better alignment) correlates with better phonological reasoning.
Cognate Analysis: They investigate why misalignment occurs, using the CogNet database to analyze loanwords and cognates.

C. Experiment 3: Mitigation via Fine-Tuning

Goal: Improve phonological reasoning without retraining the tokenizer or the entire model from scratch.
Method: A lightweight Instruction Tuning approach using International Phonetic Alphabet (IPA) annotations.
- They augment general QA data (OpenHermes2.5) by inserting <IPA> tags around words and providing IPA transcriptions in the answers.
- They construct specific datasets for rhyming, G2P, and syllable counting where the model is explicitly taught to reason using IPA.
- Model: Fine-tuned Llama-3.1-8B-Instruct using LoRA (Low-Rank Adaptation).

3. Key Contributions

Diagnosis of Tokenization Bias: The paper provides empirical evidence that subword tokenization systematically degrades phonological representations. It shows that while LMs do encode phonological knowledge in hidden states, standard tokenization obscures this information.
Introduction of STAD: The authors propose Syllabification-Tokenization Alignment Distance (STAD) as a quantitative diagnostic tool. They demonstrate a strong negative correlation between high STAD and poor phonological performance, offering a concrete metric for evaluating tokenizer quality regarding phonology.
Cognate/Lexical Source of Error: They identify that words with high cross-linguistic variation (cognates and loanwords) are more likely to be tokenized in ways that misalign with syllables, explaining why certain words are harder for models to process phonologically.
Practical Mitigation Strategy: They propose a IPA-augmented fine-tuning method. This approach successfully injects phonological awareness into existing models without modifying the vocabulary or tokenizer, serving as a practical "patch" for current LLMs.

4. Key Results

Probing Results: All tested LMs encode phonological features in their hidden states (probes significantly outperform random baselines), with the strongest signals appearing in middle layers (20%–60% depth). However, performance is far from perfect.
Tokenization Granularity: Inserting delimiters (slash tokenization) significantly improved Rhyming Awareness accuracy across all models (e.g., Llama-3.1 improved from ~72% to ~85% in middle layers), proving that coarse tokens hinder local feature extraction.
STAD Correlation: Words with low STAD (aligned tokens/syllables) yielded significantly better performance in G2P and Syllable Counting tasks compared to high-STAD words. For example, in Syllable Counting, aligned words achieved $R^2 \approx 0.98$ while misaligned words dropped to $R^2 \approx 0.85$ .
Fine-Tuning Success: The IPA-augmented Llama-3.1-8B-IPA model showed consistent improvements across all three phonology tasks:
- Rhyming: Improved accuracy.
- G2P: Reduced Phoneme Error Rate (PER).
- Syllable Counting: Improved accuracy.
General Reasoning Trade-off: The fine-tuned model retained most of its general capabilities, with only minor drops on standard benchmarks: -1.1% on GSM8K (math) and -0.9% on MMLU (general knowledge), indicating no catastrophic forgetting.

5. Significance and Future Directions

Theoretical Insight: The paper challenges the notion that text-only LMs lack phonological understanding. Instead, it argues that the interface between tokenization and phonology is the bottleneck.
Diagnostic Tool: STAD offers a simple, language-agnostic metric for tokenizer developers to evaluate how well their segmentation schemes respect phonological boundaries.
Practical Application: The proposed fine-tuning method offers a low-cost, high-impact solution for improving phonological tasks (e.g., poetry generation, language learning assistants, TTS) without the computational expense of retraining foundation models or redesigning tokenizers.
Future Work: The authors suggest that future tokenizer designs should explicitly incorporate phonological constraints (e.g., syllable-aware tokenization) and that these findings should inform the design of joint speech-text models.

In summary, this work bridges the gap between linguistic theory and deep learning engineering, demonstrating that how we split text into tokens fundamentally limits what language models can "hear" and "say," and providing a roadmap to fix it.