💬 NLP

Zonkey: A Hierarchical Diffusion Language Model with Differentiable Tokenization and Probabilistic Attention

Zonkey is a novel hierarchical diffusion language model that introduces a fully trainable pipeline from raw characters to document-level representations by utilizing a differentiable tokenizer and probabilistic attention mechanism to enable end-to-end optimization and adaptive, linguistically meaningful text generation without fixed tokenizers.

Original authors: Alon Rozental

Published 2026-01-30

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Alon Rozental

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to write a story. Usually, we teach robots by giving them a fixed dictionary of "chunks" (like words or syllables) and telling them, "If you see a space, start a new word." This is like giving the robot a rigid set of LEGO bricks that can only snap together in one specific way. If the robot encounters a new type of brick or a messy pile of sand, it gets confused because its rules don't fit.

The paper introduces Zonkey, a new kind of robot that doesn't use a fixed dictionary. Instead, it learns to build its own "chunks" as it goes, starting from individual letters and figuring out where words and sentences naturally begin and end. It does this using a special "diffusion" process, which is like sculpting a statue out of a block of marble that is slowly turning into dust and then back into stone.

Here is how Zonkey works, broken down into simple parts:

1. The "Probabilistic Attention" (The Soft Filter)

Traditional robots look at a sentence and say, "This part exists, and this part doesn't." They use hard cuts. Zonkey is more like a dimmer switch. It assigns a "probability of existence" to every part of the text.

The Analogy: Imagine looking at a foggy window. Some parts are clear (high probability), and some are very foggy (low probability). Zonkey doesn't just cut off the foggy parts; it learns to look through the fog. It understands that a word might be "mostly there" or "barely there," allowing it to handle sentences of any length without getting stuck on a fixed size.

2. The "Segment Splitter" (The Self-Learning Censor)

Usually, a computer needs a human to tell it, "Put a space here to start a new word." Zonkey has a special module called the Segment Splitter that learns this on its own.

The Analogy: Think of a long, unbroken string of letters like thequickbrownfox. A normal computer needs a rulebook to know where to cut. Zonkey is like a smart scissors that learns, "Hey, I feel like cutting after 'the' because that makes a better word." It doesn't need a rulebook; it learns to cut at spaces or periods because doing so helps it reconstruct the story better later. It discovers the concept of "words" and "sentences" by itself.

3. The "Compressor" and "Diffusion" (The Summarizer and Sculptor)

Once Zonkey cuts the text into chunks, it squeezes each chunk into a tiny, compact summary (a vector).

The Analogy: Imagine taking a whole paragraph and shrinking it down into a single, dense marble.
The Diffusion Part: To generate text, Zonkey starts with a block of pure noise (static). It then slowly "denoises" this block, turning the static into a clear image (or text).
The Twist: Most diffusion models struggle with text because text is discrete (you can't have half a letter). Zonkey uses a Mixed Model (DDMM). It's like a sculptor who knows when to take small, careful steps to get the details right, and when to take big, bold leaps to get the shape of the whole body. This helps it create long, coherent sentences without getting lost.

4. The "Stitcher" (The Seamstress)

Because Zonkey cuts the text into overlapping chunks to process them, it has to put them back together.

The Analogy: Imagine sewing two pieces of fabric together. A normal robot might just tape them side-by-side, leaving a visible, ugly seam. Zonkey's Stitcher is a master seamstress. It looks at where the two pieces overlap and blends them together so smoothly that you can't tell where one ends and the other begins. If one piece has a mistake (like saying "classical mechanics" instead of "quantum mechanics"), the Stitcher uses the overlapping piece to gently correct the error before sewing them together.

5. The Result: A Fully Trainable Robot

The biggest claim of the paper is that everything in Zonkey is "differentiable."

The Analogy: In normal robots, if you want to change how it cuts words, you have to manually rewrite the code. In Zonkey, the whole system is like a single, giant, stretchy rubber band. If the robot makes a mistake, the "error" ripples all the way back through the Stitcher, the Diffusion process, the Compressor, and all the way back to the Splitter. This tells the Splitter, "Hey, you cut in the wrong place; try cutting somewhere else next time."

What Zonkey Can Do (According to the Paper)

Generate Text from Noise: It can start with random static and slowly turn it into coherent sentences about Wikipedia topics.
Fill in the Blanks: If you give it a sentence with a missing middle part (e.g., "Bob is a [BLANK] player in the NBA"), it can fill in the gap ("basketball") without needing to write the sentence from left to right. It can fix the middle while keeping the start and end fixed.
Adapt to Any Data: Because it learns its own "words" and "sentences," it can handle messy or unusual text better than robots with fixed dictionaries.

What It Cannot Do (Yet)

The paper is very honest about its limits. Currently, Zonkey is a "proof of concept" trained on a single computer using Wikipedia data.

It can write coherent sentences, but it hasn't been tested on writing whole books or complex documents yet.
It doesn't have a fixed vocabulary, so standard tests used to measure other AI models (like counting "perplexity") don't work for it.
The paper does not claim it can do medical diagnosis, legal advice, or any real-world application yet; it is purely a research step toward a new type of language model.

In short, Zonkey is an experiment to see if we can build a language model that learns to read, write, and understand the structure of language entirely on its own, from the bottom up, without needing a human to define the rules first.

Technical Summary: Zonkey

Problem Statement

Large Language Models (LLMs) currently rely on fixed, non-differentiable tokenizers (e.g., Byte Pair Encoding) that create bottlenecks in end-to-end optimization. These static schemes struggle with out-of-vocabulary (OOV) issues, noisy data, and domain adaptation. Furthermore, existing hierarchical models often assume fixed-length segments, limiting flexibility. While diffusion models have shown promise in image generation, their application to text is hindered by discrete token constraints, semantic distortions from noise, and difficulties in handling variable-length sequences without explicit end-of-sequence (EOS) tokens.

Methodology

Zonkey proposes a fully differentiable, hierarchical diffusion framework that operates from raw characters to document-level representations. The architecture integrates five core components:

1. Probabilistic Attention

To handle theoretically infinite sequences and variable lengths without hard masking (which disrupts gradients), Zonkey introduces Probabilistic Attention.

Mechanism: Each position $k$ is assigned an existence probability $p_k \in (0, 1]$ . The attention score is modulated by a scale factor based on the ratio of existence probabilities ( $p_k/p_q$ ) for future positions.
Effect: This creates a "soft mask" where low-probability positions exert minimal influence on high-probability ones. It allows the model to simulate truncation and handle uncertainty while preserving gradient flow, enabling the modeling of theoretically infinite sequences.

2. The Segment Splitter (Differentiable Tokenizer)

Replacing static tokenizers, the Segment Splitter learns probabilistic Beginning-of-Sequence (BOS) decisions directly from data.

Process: It processes input vectors (e.g., character embeddings) via a lightweight transformer encoder to predict $p_{BOS}$ for each position.
Differentiability: Instead of hard sampling, the model uses these probabilities to modulate attention weights in downstream modules. Suboptimal splits (those that hinder reconstruction) incur higher losses, creating an implicit gradient signal that encourages the splitter to learn linguistically meaningful boundaries (e.g., spaces for words, periods for sentences) without explicit supervision.
Output: Overlapping segments with associated existence probabilities and "existence shares" (normalized weights) to ensure uniform loss contribution across overlapping regions.

3. The Compressor

This module distills variable-length segments into fixed-dimensional vector representations for higher-level abstraction.

Architecture: Learnable CLS vectors are prepended to segments, which are then processed by a Transformer Encoder using Probabilistic Attention.
Objective: The compressed vectors are optimized via a Masked Language Modeling (MLM) loss (using contrastive objectives) and reconstruction losses. This ensures that the compressed latent space captures semantic meaning rather than just surface forms, clustering similar concepts (e.g., "excellent" and "great") while penalizing arbitrary splits that increase entropy.

4. Denoising Diffusion Mixed Model (DDMM)

Zonkey employs a hybrid diffusion strategy to reconstruct segments from compressed latents.

Hybrid Approach: It combines the variance-exploring stochasticity of DDPM (small steps, high diversity) with the trajectory efficiency of DDIM (large leaps, deterministic).
Mixed-Step Objective: The model is trained to take "ambitious leaps" toward clean data when the signal is strong, but revert to "cautious small steps" when uncertain. This is achieved by minimizing the cosine distance between a denoised state and its projection on the line segment connecting the clean and heavily noised states.
Process: The model inverts compressions into lower-level vectors, predicts existence probabilities to determine sequence length, and refines the sequence using Probabilistic Attention.

5. The Segment Stitcher

To reassemble overlapping, denoised segments into coherent documents, Zonkey uses a differentiable Stitcher.

Function: It aligns consecutive segments by inferring soft offsets based on existence probability decay and content similarity.
Refinement: A lightweight cross-attention mechanism blends overlapping regions, correcting residual errors (e.g., fixing "classical mechanics" to "quantum mechanics" if the overlap suggests the latter) while maintaining global consistency. This ensures that hierarchical outputs are invariant to the specific segmentation boundaries.

Key Contributions

Fully Differentiable Pipeline: Zonkey eliminates the non-differentiable bottleneck of traditional tokenization by learning adaptive splits (Segment Splitter) and reassembly (Stitcher) end-to-end.
Probabilistic Attention: A novel attention mechanism that handles variable-length sequences and infinite contexts via soft masking, enabling gradient flow through uncertain segment boundaries.
Hierarchical Diffusion: A framework that compresses raw characters into word-like and sentence-like abstractions, reconstructing them via DDMM to generate coherent, variable-length text from noise.
Emergent Linguistic Structure: The model demonstrates the ability to learn meaningful token boundaries (spaces, periods) and hierarchical abstractions purely through reconstruction and MLM objectives, without explicit linguistic supervision.

Results

Trained end-to-end on Wikipedia data using a single GPU, Zonkey demonstrates the following capabilities:

Coherent Generation: It generates coherent, variable-length text from noise at the sentence level.
Emergent Hierarchies: The model spontaneously develops word-level and sentence-level structures, with the Segment Splitter learning to place BOS tokens at linguistically appropriate locations (e.g., after spaces or periods).
Adaptive Tokenization: Unlike fixed tokenizers, Zonkey adapts its segmentation based on the data distribution, handling noisy or domain-specific text more robustly.
Infilling: The architecture naturally supports non-sequential infilling of missing text sections by conditioning on fixed prefix/suffix vectors and denoising the gap.

Significance and Claims

The paper positions Zonkey as a significant step toward fully gradient-based language models. By removing fixed tokenizers and hard length constraints, Zonkey addresses the rigidity of current LLMs, offering potential for:

Better Domain Adaptation: The ability to learn tokenization and hierarchy directly from data allows for more robust handling of noisy or specialized domains.
Scalable Generation: The parallel processing of hierarchical levels (as opposed to sequential autoregressive generation) offers speed advantages for long-form synthesis.
Unified Optimization: The framework unifies tokenization, compression, and generation into a single differentiable objective, aligning low-level representations with high-level reconstruction quality.

The authors note that current results are preliminary (proof-of-concept on a single GPU) and that quantitative evaluation remains challenging due to the lack of a fixed vocabulary. However, the qualitative emergence of linguistic structure and the successful demonstration of variable-length diffusion generation suggest a viable path toward scalable, adaptable, and truly differentiable language modeling.