HeceTokenizer: A Syllable-Based Tokenization Approach for Turkish Retrieval

1. Problem Statement

The paper addresses the challenges of tokenizing Turkish, a highly agglutinative language where a single root word can generate hundreds of surface forms through suffix concatenation.

• Limitations of Current Methods: Standard frequency-based tokenizers (e.g., BPE, WordPiece) often split words at statistically convenient but linguistically arbitrary boundaries. This obscures semantic and grammatical structures encoded in suffixes and frequently leads to Out-of-Vocabulary (OOV) tokens for unseen surface forms.
• Limitations of Morphological Approaches: While morphology-driven tokenizers (e.g., TurkishTokenizer) improve alignment by using curated dictionaries, they are computationally expensive, require extensive lexical resources, and often necessitate large model architectures (e.g., 300M parameters) to achieve high performance.
• The Gap: There is a need for a tokenization strategy that is linguistically principled, computationally lightweight, OOV-free, and effective for retrieval tasks without relying on massive dictionaries or models.

2. Methodology

The authors propose HeceTokenizer, a syllable-based tokenization approach that leverages the deterministic phonological structure of Turkish.

A. Theoretical Basis: Deterministic Syllable Structure

Turkish syllables follow a closed set of six canonical phonological patterns (where V = vowel, C = consonant):

1. V
2. CV
3. VC
4. CVC
5. VCC
6. CVCC

Because these patterns are exhaustive and deterministic, any Turkish word (including loanwords) can be decomposed into syllables without a lexicon. This results in a closed vocabulary of approximately 8,000 unique syllable types, theoretically eliminating OOV tokens.

B. Algorithm Implementation

• Syllabification: The algorithm scans words from right to left, greedily matching the longest valid pattern at each step (Algorithm 1).
• Handling Edge Cases: Isolated consonants (common in loanwords like tren or strateji) are handled gracefully as single tokens.
• Vocabulary Construction: Applied to a large Turkish corpus, yielding a fixed vocabulary of ~8,000 tokens.

C. Model Training

• Architecture: A BERT-tiny encoder trained from scratch.
  • Parameters: 1.5M (Hidden size: 128, 2 transformer layers, 2 attention heads).
  • Objective: Masked Language Modeling (MLM) with a 15% masking rate.
• Data: 26,000 articles from Turkish Wikipedia (~40% of the corpus).
• Preprocessing: Text is syllabified and split into overlapping windows of 256 tokens (stride 128).

D. Retrieval Strategy

• Benchmark: TQuAD (Turkish Question Answering) retrieval benchmark.
• Metric: Recall@5 (fraction of questions where the correct passage is in the top 5 retrieved results).
• Chunk-Based Retrieval: Instead of retrieving full passages, the system segments passages into fine-grained overlapping chunks of syllable tokens. Each chunk is embedded independently, and the top 5 chunks are retrieved via cosine similarity.

3. Key Contributions

1. Novel Tokenization Paradigm: Introduction of a syllable-based tokenizer for Turkish that relies on phonological regularity rather than morphological dictionaries, ensuring a 100% OOV-free vocabulary.
2. Resource Efficiency: Demonstration that a 1.5M parameter model using syllable tokens can outperform 300M parameter models using morphological tokenization.
3. Optimal Granularity Discovery: Identification that 8 syllable tokens (approx. 2.6 words) is the optimal chunk size for retrieval, balancing local co-occurrence signals with context relevance.
4. Embedding Stability: Evidence that syllable-based tokenization prevents embedding collapse (mean pairwise cosine similarity of 0.257 vs. 0.998 for BPE), providing a stable training signal.

4. Results

The paper compares HeceTokenizer against several baselines on the TQuAD benchmark:

Method	Model Size	Tokenizer Type	Recall@5
HeceTokenizer (Optimal)	1.5M	Syllable (Chunk=8)	50.3%
TurkishTokenizer [3]	300M	Morphological	46.92%
Mursit [3]	300M	BPE	35.43%
CosmosGPT2 [3]	300M	BPE	33.81%
Tabi [3]	300M	BPE	34.96%

• Performance Gain: HeceTokenizer achieves a 3.38% absolute improvement over the state-of-the-art morphology-driven baseline (TurkishTokenizer) while using a model 200 times smaller.
• Chunk Size Analysis:
  • Full passage (512 tokens): 37.9% Recall@5.
  • Optimal chunk (8 tokens): 50.3% Recall@5.
  • Very small chunks (4-6 tokens): Performance drops slightly, suggesting a loss of local context.

5. Significance and Implications

• Inductive Bias: The results suggest that the phonological regularity of Turkish provides a strong inductive bias for NLP tasks. The deterministic nature of syllables allows lightweight models to learn robust representations without the overhead of morphological analysis.
• Scalability: The approach demonstrates that for agglutinative languages, "smaller is better" if the tokenization strategy is linguistically aligned. It challenges the trend of scaling model parameters to overcome poor tokenization.
• Practicality: The tokenizer requires no external dictionaries or complex morphological analyzers, making it easy to implement and deploy for any Turkish text.
• Future Directions: The authors propose extending this approach to broader benchmarks (TR-MTEB), generative tasks, and other languages with regular syllable inventories.

Conclusion

HeceTokenizer proves that exploiting the deterministic phonological structure of Turkish allows for a highly efficient, OOV-free tokenization strategy. When combined with fine-grained chunk-based retrieval, it enables a tiny (1.5M) model to significantly outperform massive (300M) morphology-driven baselines, offering a new, resource-light paradigm for Turkish NLP.