💻 computer science

Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish

Morpheus is a novel neural tokenizer and word embedder for Turkish that utilizes a differentiable Poisson-binomial dynamic program to achieve lossless, morphology-aware tokenization with superior compression efficiency and morphological alignment compared to standard subword methods, while also generating high-quality root-centric word embeddings.

Original authors: Tolga Şakar

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tolga Şakar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Lego" Language

Imagine the Turkish language is like a giant set of Mega Bloks (or Lego). In English, you usually build with whole bricks (words). But in Turkish, you start with one base brick (the root, like "house") and snap on smaller, specific pieces (suffixes) to change the meaning. You can snap on "our," "in," and "the ones who are" to turn "house" into "the ones who are in our houses."

The problem is that the AI computers we use today (Large Language Models) are used to English-style bricks. When they try to read Turkish, they use a "statistical guesser" to chop words up.

The Mistake: They chop the word in the wrong place, breaking the meaning.
The Glitch: Some of these choppy tools are "lossy." It's like they take your Lego model, snap it apart, and then try to put it back together, but they accidentally paint over the colors or swap a red brick for a blue one. When the computer tries to speak back, it says the wrong thing.

The Solution: Morpheus

The author, Tolga Şakar, built a new tool called Morpheus. Think of Morpheus as a smart, magical pair of scissors that understands exactly where the Lego pieces connect.

Here is how it works, broken down into three simple superpowers:

1. The Perfect Chopper (Lossless Tokenizer)

Most tools chop words based on how often they appear in a book. Morpheus chops them based on grammar.

The Analogy: Imagine a chef cutting a cake. A normal chef cuts it into random slices to make them look equal. Morpheus is a chef who knows exactly where the layers of chocolate and vanilla are, so they cut only between the layers.
The Result: It never breaks a word in a way that changes its spelling. If you give it a word, chop it up, and then put it back together, you get the exact same word back, down to the last dot and accent. This is crucial because if an AI is writing a story, it needs to be able to spell the words correctly when it talks.

2. The Instant Translator (Word Embedder)

Usually, you need two different machines to do two jobs: one to chop the words, and a second, huge machine to understand what those words mean.

The Analogy: Usually, you have to hire a Librarian to find the book (chop the word) and then a Professor to explain the story (understand the meaning).
Morpheus's Trick: Morpheus is a Librarian-Professor hybrid. The moment it chops the word, it instantly knows the meaning. It creates a "fingerprint" (an embedding) for the word at the same time it cuts it. It does this in one single step, saving time and computer memory.

3. The Root-Seeker (Smart Geometry)

Because Turkish words change so much (adding "our," "in," "s"), Morpheus is trained to ignore the changing parts and focus on the root.

The Analogy: Imagine a family tree. "House," "Houses," "My House," and "Our Houses" are all different people, but they are all from the same family.
Morpheus's View: Morpheus sees all these variations as the same family. It groups them tightly together in its brain. This makes it amazing at finding related words (like a super-accurate search engine for roots), but it's less good at noticing the tiny differences between "My House" and "Your House" if the sentence context is complex.

The Trade-Offs (The "Fine Print")

The paper is honest about what Morpheus gives up to gain these powers:

More Chunks: Because it cuts words into their true grammatical parts, it creates more "chunks" (tokens) per word than the standard tools.
- Analogy: It's like sending a letter where you write every single syllable on a separate line. It takes a bit longer to write and read (slightly slower speed), but the message is much clearer and more accurate.
Specialized Brain: It is a master at understanding the roots of words, but it isn't as good at understanding the context of a whole sentence as the giant, heavy AI models (like BERT) are.
- Analogy: Morpheus is a brilliant word detective, but it's not a novelist. It's perfect for finding specific words or cleaning up data, but for writing a complex story, you might still want to pair it with a bigger, context-aware model.

The Bottom Line

Morpheus is a new tool for Turkish AI that fixes the "broken Lego" problem.

It never loses the original spelling (it's reversible).
It understands the grammar of the cuts, not just the statistics.
It gives you a meaning map for the words at the same time it cuts them.

The paper proves that for Turkish, this "smart cutter" is better than the old "statistical guessers" for tasks that require accuracy, memory efficiency, and understanding word families, even if it's slightly slower at raw speed.

Technical Summary: Morpheus – A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish

1. Problem Statement

Turkish is an agglutinative language where semantic meaning is carried by chains of derivational and inflectional suffixes attached to a root. Current Natural Language Processing (NLP) tools for Turkish face two primary limitations:

Tokenization Deficiencies: Standard subword tokenizers (BPE, WordPiece, Unigram) segment words based on corpus statistics rather than linguistic morphology. This leads to the fragmentation of semantically loaded suffixes, inflating sequence lengths (fertility), and increasing computational costs. Furthermore, many widely used tokenizers are not reversible; they strip diacritics or apply canonical re-harmonization (normalization), meaning decode(encode(w)) does not equal the original string w. This loss of surface fidelity corrupts generative model outputs. Conversely, rule-based analyzers (e.g., Zemberek) and unsupervised segmenters (e.g., Morfessor) offer better morphological alignment but are either lossy, dictionary-bound, or not optimized for language modeling.
Representation Gaps: Current word representation systems treat tokenization and embedding as separate, disjoint processes. Contextual encoders like BERTurk are heavy, rely on lossy subword vocabularies, and treat morphology only implicitly. There is a lack of a unified system that produces discrete, lossless tokens while simultaneously generating structured word embeddings that inherently respect morphological relationships (e.g., grouping kitap, kitaplar, kitabımız by construction).

2. Methodology

The paper introduces Morpheus, a neural morpheme-boundary model designed to function simultaneously as a lossless, morphology-aware tokenizer and a word-embedding producer.

Architecture

Morpheus processes a word as a character sequence and outputs both morpheme boundaries and a single word embedding in a single forward pass:

Character Encoder: Character embeddings are concatenated with learned case-flag embeddings and passed through multi-scale convolutions (kernel widths 2–6) and three self-attention layers. Crucially, Rotary Position Embeddings (RoPE) are applied to capture relative offsets between characters, allowing the model to reason about suffix positions relative to the root rather than absolute indices.
Boundary Detector: A stack of four RoPE self-attention layers followed by a scoring head predicts a probability $p_i \in [0, 1]$ for a boundary at each inter-character position.
Differentiable Poisson–Binomial Segmentation: To avoid non-differentiable argmax operations that would block gradient flow, Morpheus employs a differentiable dynamic program. It treats boundary probabilities as independent Bernoulli trials and computes a soft morpheme membership matrix $M$ $M$ using a Poisson–binomial distribution.
- Training: Gradients flow through the soft membership matrix $M$ to the boundary detector, shaped by both supervised labels and self-supervised objectives.
- Inference: As probabilities converge to 0 or 1, the soft matrix becomes a one-hot hard segmentation, recovering exact boundaries without architectural changes.
- Surface Preservation: The mechanism only groups characters; it never inserts, drops, or rewrites them. This ensures decode(encode(w)) = w holds by construction.
Word Embedding: Segment vectors are computed via attention-pooling of character vectors weighted by their membership in $M$ . These are aggregated into a final 320-dimensional word embedding ( $e_w$ ) via a feed-forward network.

Training Objectives

The model is trained with a weighted sum of four losses:

Auxiliary Loss ( $L_{aux}$ ): A boundary Binary Cross-Entropy (BCE) and count regularizer against labels from an unsupervised Morfessor analyzer (with root-correction). This weight decays curricularly from 0.50 to 0.08.
Skip-Gram Negative Sampling ( $L_{sgns}$ ): Standard word2vec-style objective.
Contrastive Loss ( $L_{ctr}$ ): An InfoNCE loss on root identity, pulling all inflections of the same root together.
Masked Language Modeling ( $L_{mlm}$ ): A vocabulary-free, character-level reconstruction task where masked words are regenerated character-by-character.

3. Key Contributions

Unified Model: Morpheus is the first model to act as both a lossless, morphology-aware tokenizer and a structured word-embedding producer for Turkish, unifying tokenization and representation.
Differentiable Segmentation: The introduction of a differentiable Poisson–binomial dynamic program allows for end-to-end training of boundary detection while guaranteeing exact, reversible hard segmentation at inference.
Comprehensive Evaluation: The paper establishes a new evaluation axis for Turkish tokenization: reversibility. It demonstrates that many high-performing tokenizers are invalid for generative tasks due to non-reversible decoding.

4. Results

Tokenization Performance

Reversibility: Morpheus achieves 100% reversibility (decode(encode(w)) = w), whereas WordPiece (58.2%) and the rule-based TurkishTokenizer (95.4%) fail due to diacritic stripping and canonical normalization, respectively.
Morphological Alignment: On the MorphScore benchmark (UD Turkish-Kenet), Morpheus achieves a macro-F1 of 0.61, roughly double that of subword tokenizers (~0.32) and comparable to the rule-based TurkishTokenizer (0.65), but without the lossy decoding caveats.
Efficiency & Quality: Among reversible tokenizers, Morpheus attains the lowest Bits-Per-Character (BPC) of 1.425 (vs. 1.436 for BPE). It uses approximately 19% less peak GPU memory than 64K-vocabulary subword tokenizers during generation.
Fertility: Morpheus has a higher token-per-word ratio (1.73) compared to subword tokenizers (~1.5), a trade-off for morpheme-level granularity.

Embedding Performance

Evaluated on frozen vectors against BERTurk and BGE-M3:

Lexical/Root Tasks: Morpheus leads decisively in root-family retrieval (MAP 0.85 vs. 0.80 for BGE-M3) and same-root verification (ROC-AUC 1.00 vs. 0.98). The root-centric geometry effectively clusters inflections.
Contextual/Inflectional Tasks: Morpheus underperforms on tasks requiring sentence context or fine-grained inflectional distinctions (e.g., NER, number/case probing), where heavier contextual encoders (BERTurk, BGE-M3) remain superior.

5. Significance and Claims

The paper argues that reversibility is the decisive gate for tokenizers intended for generative models. By filtering out non-reversible tokenizers, Morpheus emerges as the superior option for Turkish NLP, offering the lowest BPC, highest morphological alignment, and lowest memory footprint simultaneously.

The authors position Morpheus not as a replacement for contextual encoders, but as a complementary, morphology-aware lexical encoder. Its root-centric embedding is ideal for lexical retrieval, deduplication, and keyword matching in multi-vector RAG systems, while contextual encoders handle sentence-level semantics.

The work demonstrates that a single neural mechanism can successfully bridge the gap between tokenization and representation, providing a "lossless, morphology-aware, embedding-producing" alternative to the fragmented, lossy, or morphology-blind tools currently available for agglutinative languages.