💬 NLP

Esoteric Language Models: A Family of Any-Order Diffusion LLMs

This paper introduces Eso-LMs, a novel family of diffusion language models that fuse autoregressive and masked diffusion paradigms using causal attention to enable KV caching and parallel generation, thereby establishing a new state-of-the-art on the speed-quality Pareto frontier for unconditional generation.

Original authors: Subham Sekhar Sahoo, Zhihan Yang, Yash Akhauri, Johnna Liu, Deepansha Singh, Zhoujun Cheng, Zhengzhong Liu, Eric Xing, John Thickstun, Arash Vahdat

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Subham Sekhar Sahoo, Zhihan Yang, Yash Akhauri, Johnna Liu, Deepansha Singh, Zhoujun Cheng, Zhengzhong Liu, Eric Xing, John Thickstun, Arash Vahdat

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Two Ways to Write a Story

Imagine you are trying to write a sentence, word by word. Currently, AI models generally use one of two methods to do this:

The "Strict Line" Method (Autoregressive or AR):
Think of this like a person writing a letter with a pen. They must write the first word, then the second, then the third. They cannot skip ahead. If they want to change the first word, they have to erase and rewrite everything after it.
- Pros: It’s very accurate and coherent.
- Cons: It’s slow. You have to wait for every single word to be written before you can see the next one.
The "Blindfolded Sketch" Method (Masked Diffusion or MDM):
Imagine you are given a page full of blank boxes (masks). The AI’s job is to fill in these boxes with words. But instead of filling them in one by one, it looks at the whole page at once and tries to guess what words might fit in any of the empty boxes, all at the same time. It does this in several rounds, slowly revealing more words until the sentence is complete.
- Pros: It can work on many words in parallel (at the same time), which should be faster.
- Cons: In practice, it’s actually slower than the "Strict Line" method because the computer has to re-calculate the whole page every time it reveals a new word. It also doesn’t have a "memory shortcut" (called KV Caching) that helps the "Strict Line" method run fast.

The Problem

The "Blindfolded Sketch" method (MDM) is promising because it allows for more control and parallel processing, but it has two major flaws:

It’s sluggish: Because it lacks the "memory shortcut," it wastes computing power re-doing work it already did.
It’s less accurate: It doesn’t understand the flow of language as well as the "Strict Line" method.

Recent attempts to fix this (like a method called BD3-LMs) tried to break the sentence into small blocks and fill them in one by one. But this caused new problems: if you rushed the process, the sentences became nonsensical or repetitive.

The Solution: "Esoteric Language Models" (Eso-LMs)

The authors created a new hybrid model called Eso-LMs. Think of it as a "Smart Hybrid Car" that can switch between two engines depending on the road conditions.

1. The Hybrid Approach

Eso-LMs combines the best of both worlds:

Phase 1 (The Sketch): It starts like the "Blindfolded Sketch" method, filling in some words in parallel across the whole sentence.
Phase 2 (The Line): Once some words are filled in, it switches to the "Strict Line" method to fill in the remaining gaps from left to right.

By adjusting a dial (called $\alpha_0$ ), you can decide how much of the sentence is done in parallel and how much is done sequentially. This lets the model smoothly slide between being a pure "Sketch" model and a pure "Line" model.

2. The "Memory Shortcut" (KV Caching)

The biggest breakthrough is that Eso-LMs finally allows the "Blindfolded Sketch" method to use the memory shortcut (KV Caching).

Analogy: Imagine you are painting a mural. In the old method, every time you added a new brushstroke, you had to repaint the entire background to make sure the colors matched. In Eso-LMs, once you paint a section, you "lock" it in your memory. When you paint the next section, you only look at the locked sections and the new area you are working on. You don’t repaint the whole wall.
Result: This makes the model significantly faster, especially for long texts.

3. "Any-Order" Attention

To make this work, the authors changed how the AI "looks" at the words.

Standard "Sketch" models look at all words equally (bidirectional).
Standard "Line" models only look at previous words (causal).
Eso-LMs uses a clever trick: It shuffles the words so that the "filled-in" words come first, and the "empty" words come last. It then uses the "Line" method’s logic on this shuffled list. This allows it to use the memory shortcut while still filling in words in a random, parallel order.

Why Is This Important?

Speed vs. Quality Balance: The paper shows that Eso-LMs sits on the "Pareto Frontier." This is a fancy way of saying it offers the best possible trade-off between speed and quality. It is faster than the old "Sketch" models and produces higher-quality text than the rushed "Block" methods.
No "Degenerate" Samples: Unlike previous hybrid methods that produced gibberish when rushed, Eso-LMs maintains good quality even when you ask it to generate text very quickly (using fewer steps).
Exact Likelihood: For the first time, the authors provide a way to calculate the exact "probability" (likelihood) of the text generated by these diffusion models. This is important for training the AI to reason better using reinforcement learning.

In Summary

Eso-LMs is a new type of AI language model that acts like a hybrid vehicle. It starts by filling in words in parallel (like a sketch) and finishes by filling in the rest in order (like a line). By using a clever shuffling trick, it unlocks a "memory shortcut" that makes it much faster than previous parallel models, without sacrificing the quality of the text. It’s faster, more flexible, and more efficient than the current state-of-the-art methods.

Technical Summary: Esoteric Language Models (Eso-LMs)

1. Problem Statement
The paper addresses the limitations of existing Masked Diffusion Models (MDMs) in language modeling. While MDMs offer parallel and controllable generation, they currently underperform Autoregressive (AR) models in perplexity and lack critical inference-time efficiency features, most notably Key-Value (KV) caching. Existing attempts to bridge this gap, such as Block Diffusion Language Models (BD3-LMs), introduce semi-autoregressive strategies that partially support KV caching but suffer from severe sample quality degradation at low sampling steps (low Number of Function Evaluations, or NFEs) and incomplete KV caching within blocks. The core challenge is to fuse the parallel generation capabilities of MDMs with the efficiency and likelihood accuracy of AR models without sacrificing generation quality.

2. Methodology
The authors propose Esoteric Language Models (Eso-LMs), a hybrid framework that interpolates between AR and MDM paradigms. The methodology rests on three pillars:

Hybrid Generative Process: Eso-LMs decompose generation into two phases:
1. Diffusion Phase: An MDM component generates a partially masked sequence in parallel.
2. Sequential Phase: An AR component fills in the remaining masked tokens sequentially from left to right.
  The model is trained with a hybrid loss function combining AR and MDM objectives, allowing smooth interpolation between the two paradigms via a hyperparameter $\alpha_0$ , which controls the expected fraction of tokens generated by the diffusion phase.
Causal Attention and KV Caching: Unlike prior MDMs that use bidirectional attention, Eso-LMs exploit the connection between MDMs and "Any-Order" autoregressive models by adopting causal attention. This design enables:
- Exact KV Caching: By ensuring previously predicted tokens do not depend on future tokens to be denoised, the model supports exact KV caching during the diffusion phase.
- Parallel Generation: The model preserves parallel generation over the full sequence length during the diffusion phase.
- Efficient Forward Passes: The forward pass is restricted to clean tokens and the current mask tokens to be updated, significantly reducing computation for long sequences.
Unified Attention Mechanism: A specialized attention bias matrix is introduced to support both generation styles within a single transformer. During training, the model uses causal attention on shuffled sequences (for diffusion) and a specialized sparse attention mask on concatenated sequences (for sequential training) to ensure clean left contexts for AR-style prediction.
Exact Likelihood Estimation: The paper derives the first (asymptotically) exact likelihood formula for MDMs using an importance-weighted (IW) bound. This allows for tractable, single-pass estimation of the Negative Evidence Lower Bound (NELBO), which is crucial for reinforcement learning fine-tuning.

3. Key Contributions

New Hybrid Framework: Introduction of Eso-LMs, which outperforms previous hybrid approaches (BD3-LMs) and enables fine-grained interpolation between AR and MDM perplexities.
State-of-the-Art Pareto Frontier: Eso-LMs establish a new state of the art on the speed-quality Pareto frontier for unconditional generation. Unlike BD3-LMs, Eso-LMs do not degrade in quality at low sampling steps.
Inference Efficiency: On long contexts, Eso-LMs provide 14–65× faster inference than standard MDMs and 3–4× faster inference than BD3-LMs.
Theoretical Advance: Derivation of the first asymptotically exact likelihood formula for MDMs, enabling efficient likelihood estimation and RL-based fine-tuning.

4. Results

Perplexity: On LM1B and OpenWebText (OWT) benchmarks, Eso-LMs smoothly interpolate between AR and MDM perplexities. While the pure diffusion mode ( $\alpha_0=1$ ) has slightly higher perplexity than MDLM due to the use of causal attention, the hybrid modes ( $\alpha_0 < 1$ ) achieve competitive perplexity close to AR models.
Speed-Quality Trade-off: Eso-LMs maintain high sample quality (measured by MAUVE and Generative Perplexity) even at low NFEs, whereas BD3-LMs exhibit sharp quality drops. Eso-LMs set a new state of the art on the speed-quality Pareto frontier.
Long Context Latency: For context lengths of 2048 to 10240, Eso-LMs are significantly faster than MDLMs (up to 65×) and BD3-LMs (up to 3.8×) due to efficient KV caching and restricted forward passes.
Likelihood Estimation: The single-pass NELBO estimator for Eso-LMs shows lower variance than standard Monte Carlo estimators for MDMs. The exact likelihood estimation confirms that Eso-LMs ( $\alpha_0=1$ ) have true perplexity comparable to MDLM’s NELBO perplexity.

5. Significance
The paper claims that Eso-LMs represent a paradigm shift by successfully fusing AR and MDM strengths. The significance lies in overcoming the primary limitations of MDMs—lack of KV caching and inference inefficiency—while preserving their parallel generation capabilities. By enabling exact KV caching and parallel generation, Eso-LMs achieve superior inference efficiency and robust generation quality across varying sampling budgets, establishing a new benchmark for unconditional language generation. The work also provides a theoretical foundation for exact likelihood estimation in diffusion models, facilitating future applications in reinforcement learning.