💻 computer science

Sumi: Open Uniform Diffusion Language Model from Scratch

The paper introduces Sumi, a fully open 7B uniform diffusion language model pretrained from scratch on 1.5 trillion tokens, which serves as the first large-scale reference point for studying native uniform diffusion and demonstrates competitive performance on knowledge, reasoning, and coding benchmarks.

Original authors: Mengyu Ye, Keito Kudo, Wataru Ikeda, Ryosuke Matsuda, Keisuke Sakaguchi, Jun Suzuki

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mengyu Ye, Keito Kudo, Wataru Ikeda, Ryosuke Matsuda, Keisuke Sakaguchi, Jun Suzuki

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: A New Way to Write with AI

Imagine you are trying to write a story.

The Old Way (Autoregressive Models): This is like writing with a pen on paper. You write one word, then the next, then the next. Once you write a word, it's done. If you make a mistake, you have to cross it out or start the whole page over. You can't go back and change the first word without rewriting everything that came after it.
The New Way (Sumi / Uniform Diffusion): Imagine you are painting a picture on a canvas that is already covered in static noise (like TV snow). Instead of writing word-by-word, you look at the whole canvas at once. You can pick any word in the sentence, erase it, and replace it with a better one. You can do this over and over, refining the whole story simultaneously until it's perfect.

The paper introduces Sumi (which means "ink" in Japanese), a 7-billion-parameter AI model that uses this "painting" method. It is the first time anyone has built a model this big from scratch using this specific technique, rather than just tweaking an old "pen-writing" model.

How They Built It: The "School" Analogy

To train Sumi, the researchers didn't just dump random internet text into it. They curated a very specific "school curriculum."

The Textbook (Pre-training): They fed the model 1.3 trillion words. But they didn't just grab anything. They filtered the data to prioritize high-quality, educational content. Think of it as giving the AI a library full of textbooks, encyclopedias, and code manuals, while filtering out a lot of casual internet chatter, memes, or low-quality blogs.
The Specialized Training (Mid-training): After the general schooling, they gave it a "boot camp" focused on three specific subjects: Coding, Math, and Reasoning. They added even more code and math problems to its diet.
The Result: Because of this "education-heavy" diet, Sumi is very good at knowledge, logic, and writing code. However, because it didn't eat enough "casual conversation" or "common sense" food, it struggles a bit with everyday common sense questions (like "Why do people carry umbrellas when it rains?").

The Performance: How Good is the "Painter"?

The researchers tested Sumi against other famous AI models (like Llama and Falcon) that use the traditional "pen-writing" method.

The Wins: On tests involving general knowledge, reasoning, and coding, Sumi performed just as well as the best "pen-writing" models, even though those other models were trained on much more data.
The Lows: On tests requiring "common sense" (like understanding social situations), Sumi scored lower. The authors admit this is likely because their training data was too focused on school subjects and not enough on everyday life.

The "Canvas" Experiment: Finding the Sweet Spot

One of the most interesting parts of the paper is how they tested the model's "flexibility."

The Canvas Size: In this AI, you have to decide how long the sentence will be before you start generating it. This is called the "canvas length."
The Finding: Sumi works best when the canvas is exactly 2048 tokens long.
- If the canvas is too short, the model gets confused and writes gibberish.
- If the canvas is too long, it also gets confused.
- It's like a musician who can play a song perfectly if the stage is a specific size, but if the stage is too small or too big, they lose their rhythm.

The "Commitment" Mystery: How Does It Decide?

Since Sumi can change any word at any time, you might wonder: Does it just guess randomly, or does it have a plan?

The Discovery: The researchers found that Sumi doesn't have a fixed order (like "start at the beginning"). Instead, it uses a "confidence" system.
- It looks at the whole sentence and says, "I'm pretty sure about this word, so I'll lock it in. I'm not sure about that word, so I'll keep working on it."
- This creates a natural order where it locks in the easy parts first and saves the hard parts for later.
Parallel Speed: Because it locks in confident words first, it can actually write multiple words at the same time (parallel decoding) for coding tasks, which is faster than the old "one word at a time" method.

The "Self-Correction" Myth

A major promise of this "painting" method is that the AI should be able to fix its own mistakes easily. The researchers tested this by giving Sumi extra time to revise its answers.

The Result: Surprisingly, it didn't help. Even when Sumi was allowed to rewrite its answers multiple times, it mostly just changed a word and then changed it back to the original. It didn't actually "learn" to fix its errors during the generation process. It seems the model hits a ceiling and doesn't know how to improve a committed answer once it's made.

Summary

Sumi is a bold experiment showing that we can build a powerful AI from scratch using a "painting" method (Uniform Diffusion) instead of the traditional "writing" method.

It's great at: Math, coding, and facts (because it studied hard).
It's okay at: Common sense (because it skipped the "fun" stuff).
It's weird at: Changing its mind (it doesn't self-correct well yet).

The authors released the model, the code, and the exact recipe of what they fed it, hoping other scientists will use it to figure out how to make this "painting" method even better.

Technical Summary: Sumi – Open Uniform Diffusion Language Model from Scratch

Problem Statement

While diffusion models have emerged as a promising alternative to autoregressive (AR) language models, a significant gap exists in the development of Uniform Diffusion Language Models (UDLMs) at scale. Unlike Masked Diffusion Language Models (MDLMs), which fill masked tokens irreversibly, UDLMs allow any token to be updated at any step, theoretically enabling more flexible generation and self-correction.

However, no UDLM has been pretrained from scratch at both large parameter scales (billions of parameters) and large token budgets (trillions of tokens). Existing large-scale UDLMs are either compute-optimal checkpoints trained on small token budgets or models adapted from pretrained AR architectures (e.g., DiffusionGemma). Consequently, the behavior of native, large-scale uniform diffusion in the data-rich regime remains unexplored, lacking a clean reference point for studying scaling laws, generation dynamics, and trade-offs against established AR and MDLM baselines.

Methodology

Model Architecture and Training Objective

The authors introduce Sumi, a fully open 7B-parameter UDLM.

Architecture: Sumi utilizes a time-agnostic bidirectional Transformer with 36 layers, a hidden size of 4,096, and SwiGLU MLPs. It employs grouped-query attention (32 heads, 8 KV groups), RMSNorm, and untied input/output embeddings. To mitigate attention sinks, an off-by-one softmax is applied in attention.
Objective: The model is trained using the Generalized Interpolating Discrete Diffusion (GIDD) objective, specifically in its Signal-to-Noise Ratio (SNR) reparameterized form.
Noise Schedule: Training uses pure uniform noise with the log-SNR restricted to $\lambda \in [-9, 9]$ .
Tokenization: The OLMo 3 tokenizer is used with a vocabulary size of 100,278.

Training Regime

Sumi was pretrained from scratch on 1.5 trillion tokens using 288 NVIDIA H100 GPUs (totaling ~43,308 GPU-hours).

Pre-training (1.3T tokens): Conducted at a sequence length of 1,184 with a global batch size of 4,608. The data mixture is heavily weighted toward educational content, filtered and re-ranked by educational scores derived from a distilled Qwen3-32B classifier.
Mid-training (250B tokens): Two stages follow pre-training:
- 130B tokens at the original length (1,184).
- 120B tokens at an extended length (4,864).
- The mixture shifts focus toward code, math, and reasoning, comprising 32.5% coding, 29.7% math, 21.0% general, and 16.8% reasoning data.

Inference Strategy

Canvas Length: The model uses a fixed canvas length of 2,048 tokens for all evaluations. This length was chosen because performance degrades significantly on substantially shorter or longer canvases.
Sampling: The authors explore confidence-based adaptive sampling versus ancestral sampling.
Evaluation Protocol: All evaluations use the lm-evaluation-harness modified for diffusion scoring. For likelihood tasks, positions beyond the answer are filled with random tokens; for generation tasks, updates are restricted to the generation length.

Key Contributions

First Native Large-Scale UDLM: Sumi is the first UDLM pretrained from scratch at a large parameter scale (7B) and large token budget (1.5T), providing a baseline for studying native uniform diffusion.
Open Release: The authors release model weights, checkpoints, and the full training recipe, including a complete specification of the data mixture over publicly available corpora.
Exploratory Inference Probes: The paper presents directional analyses on generation dynamics, including canvas length sensitivity, token commitment order, parallel decoding capabilities, and self-correction potential.

Results

Benchmark Performance

Sumi was evaluated on 13 benchmarks across four categories and compared against open AR models (Falcon-7B, Llama 2-7B, OLMo-7B) trained on comparable token budgets.

General Knowledge & Coding: Sumi performs competitively, achieving the best scores among models evaluated under the same protocol on MMLU, TruthfulQA, HumanEval, and MBPP. This is attributed to the education- and code-heavy data mixture.
Reasoning & Mathematics: Sumi is competitive with Llama 2-7B and outperforms Falcon-7B on tasks like GSM8K and ARC, though it trails behind larger or differently trained models (e.g., LLaDA-8B, Llama 3-8B).
Commonsense: Sumi underperforms significantly on commonsense benchmarks (PIQA, HellaSwag, WinoGrande). The authors attribute this gap to the education-heavy data mixture, which has been observed to degrade commonsense performance, though they note the gap is likely larger than data composition alone can explain.

Inference-Time Observations

Canvas Sensitivity: Generation fluency is highly sensitive to canvas length. Sumi generates fluently within a task-dependent band (centered around 2,048) but degrades sharply outside this range, particularly on short canvases.
Commitment Order: While the training objective is order-agnostic, confidence-based sampling induces a self-organized commitment order. The model tends to commit determinable tokens early and defer uncertain ones, a structure not present in ancestral sampling.
Parallel Decoding: On coding tasks (HumanEval, MBPP), the model supports modest parallel decoding (committing up to 4 tokens per step) with minimal accuracy loss. However, multi-step arithmetic (GSM8K) remains order-sensitive, degrading immediately with parallel steps.
Self-Correction: An explicit revision budget (allowing extra denoising passes) does not yield self-correction. While the model overwrites committed tokens, these edits are predominantly "round trips" ( $A \to B \to A$ ) rather than directed improvements, leaving final accuracy unchanged.

Significance and Claims

The paper positions Sumi as a foundational resource for the community to study native uniform diffusion at scale. The authors emphasize that their inference-time probes are "directional rather than conclusive," intended to motivate future research into poorly understood aspects of UDLMs, such as:

The intrinsic generation behaviors of uniform diffusion versus masked diffusion.
The conditions under which self-correction might be achievable.
The impact of data composition on the specific strengths and weaknesses of diffusion models.

The authors explicitly state that Sumi is released as a pretrained base model without instruction tuning or safety filtering, intended to support research rather than direct deployment. They caution that the model inherits standard risks associated with base LMs, including potential for harmful outputs and factual inaccuracies.