🤖 machine learning

Optimality of FSQ Tokens for Continuous Diffusion for Categorical Data with Application to Text-to-Speech

This paper theoretically and empirically demonstrates that the Finite Scalar Quantization (FSQ) tokenization scheme is optimal for continuous diffusion models generating categorical data, as evidenced by its superior performance in text-to-speech tasks where it outperforms autoregressive large language models while being significantly smaller and faster.

Original authors: Vadim Popov, Wenju Gu, Tasnima Sadekova, Georgii Aparin, Assel Yermekova

Published 2026-06-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Vadim Popov, Wenju Gu, Tasnima Sadekova, Georgii Aparin, Assel Yermekova

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to speak. To do this, the robot needs to understand that human speech isn't a smooth, endless river, but rather a series of distinct building blocks (like words or sounds). In the world of AI, these blocks are called tokens.

For a long time, the best way to teach robots to generate these blocks was to use a method called "Autoregression," which is like writing a sentence one word at a time, looking back at every previous word to decide the next one. It's accurate but slow and requires a massive brain (a huge computer model) to do it.

Recently, scientists discovered a faster way called Diffusion. Think of diffusion like a sculptor starting with a block of noisy, static-filled clay and slowly chipping away the noise until a perfect statue emerges. This works great for smooth things like images or music. But trying to use it for "discrete" things like speech tokens (which are like distinct Lego bricks, not smooth clay) is tricky. The noise gets in the way, and the robot struggles to know which specific Lego brick to pick.

This paper introduces a new way to organize those Lego bricks so the diffusion process works perfectly. Here is the breakdown:

1. The Problem: The "Noisy Room"

Imagine you are in a dark room full of people (the tokens). You want to find a specific person, but the room is foggy (the noise).

The Old Way: The fog is so thick that the people are scattered randomly. It's hard to tell who is who.
The Goal: We need to arrange the people in the room so that even in the fog, the robot can easily guess who is standing where.

2. The Solution: The "Perfect Grid" (FSQ)

The authors found that the best way to arrange these people (tokens) is using a method called Finite Scalar Quantization (FSQ).

Think of FSQ as a perfectly organized grid in a 3D cube.

Instead of people standing randomly, they are placed at exact, evenly spaced coordinates (like -1, 0, and +1 on a ruler).
This creates a "grid" where every single token has a unique, predictable spot.
The paper proves mathematically that this specific grid arrangement is the optimal layout for the diffusion process. It's like arranging the Lego bricks in a perfect box so that no matter how much you shake the box (add noise), you can still easily sort them back into their original slots.

3. The Proof: Why This Grid Wins

The authors didn't just guess; they did two things to prove this grid is the best:

The Math: They showed that this grid arrangement minimizes the "confusion" (measured by something called KL divergence) between the paths the robot takes to find different tokens. In simple terms, the paths to find "Cat" and "Dog" are distinct enough that the robot doesn't get lost, even when the noise is high.
The Experiment: They built a "toy" version of the system and tried random arrangements of tokens versus the perfect FSQ grid. The FSQ grid consistently won, predicting the correct tokens much more accurately.

4. The Real-World Test: The New Voice Assistant

To prove this works in the real world, the team built a new Text-to-Speech (TTS) system (a robot that reads text aloud).

They took an existing, powerful voice system (CosyVoice2) that used a massive "brain" (a Large Language Model or LLM) to generate speech.
They replaced that massive brain with their new, smaller, diffusion-based brain that uses the FSQ grid.

The Results:

Better Quality: The new robot spoke more clearly and naturally than the old one.
Faster: Because it doesn't have to write the speech one word at a time, it is 5 to 10 times faster.
Smaller: The new model is 10 times smaller than the old one. It's like replacing a supercomputer with a smart tablet and getting better results.
Zero-Shot Cloning: Just like the old system, it could mimic a person's voice from a short sample, but it did it more efficiently.

Summary

The paper argues that if you want to use the "sculpting" (diffusion) method to generate discrete speech sounds, you must arrange your sound blocks (tokens) in a perfect, uniform grid (FSQ). This arrangement makes the math work better, the training more stable, and the final voice assistant faster, smaller, and clearer than the current giants in the field.

Technical Summary: Optimality of FSQ Tokens for Continuous Diffusion for Categorical Data with Application to Text-to-Speech

1. Problem Statement

Continuous diffusion models have achieved state-of-the-art results in continuous domains (e.g., images, video) but face significant challenges when applied to categorical data, such as text or discrete speech tokens. While discrete diffusion models exist, a hybrid framework known as Continuous Diffusion for Categorical Data (CDCD) has emerged. In CDCD, forward and reverse diffusion processes operate in a continuous latent space containing embeddings of discrete tokens, while the neural network predicts the probabilities of these discrete tokens.

However, the CDCD framework has received less attention than purely discrete or continuous approaches. A primary bottleneck is the instability of training, often attributed to the joint optimization of the latent space structure and the diffusion model. The structure of the continuous latent space is critical: if the geometry of token embeddings is suboptimal, the diffusion paths corresponding to different tokens may become indistinguishable, hindering the model's ability to learn accurate reverse trajectories. This paper investigates the geometric properties of latent spaces that make them optimally suited for CDCD models, specifically focusing on the relationship between token embedding distances and the Kullback-Leibler (KL) divergence of diffusion path measures.

2. Methodology

Theoretical Framework

The authors establish a theoretical connection between the geometry of the latent space and the training dynamics of CDCD models.

Diffusion Path Measures: The paper derives that the KL divergence between reverse-time diffusion path measures leading to two different data samples (token embeddings) $a_1$ and $a_2$ is proportional to the squared Euclidean distance $\|a_1 - a_2\|^2_2$ and a time-dependent integral of the forward diffusion coefficients.
Optimality Criterion: Based on this derivation, the authors hypothesize that maximizing the average nearest-neighbor distance between token embeddings in the latent space minimizes the KL divergence between their corresponding diffusion paths. This separation makes it easier for the neural network to distinguish between trajectories leading to different tokens, thereby improving training stability and prediction accuracy.
FSQ Analysis: The paper focuses on Finite Scalar Quantization (FSQ), a method where token embeddings are located "uniformly" in a hypercube with coordinates restricted to specific integer values. The authors analyze two specific FSQ configurations:
- Base2: Coordinates are $\{-1, 1\}$ .
- Base3: Coordinates are $\{-1, 0, 1\}$ .
Theoretical Proofs:
- Theorem 4.3: Proves that FSQ codebooks (Base2 and Base3) are locally optimal in terms of average nearest-neighbor distance among all codebooks with entries bounded by an $L_\infty$ norm of 1. Any small perturbation of an FSQ embedding reduces this distance.
- Best Accuracy Hypothesis: The authors hypothesize that for an optimally trained CDCD model with uniform token priors, the average prediction accuracy (top-1) is maximized when the latent space follows the FSQ structure. This is proven analytically for 1-dimensional latent spaces.

Experimental Validation

Numerical Experiments: The authors conducted Monte Carlo simulations to test the global optimality of FSQ codebooks. They generated thousands of random codebooks and found that FSQ configurations consistently achieved higher average nearest-neighbor distances and higher prediction accuracies compared to perturbed or random embeddings.
Text-to-Speech (TTS) Application: To validate findings in a real-world scenario, the authors replaced the autoregressive Large Language Model (LLM) backbone of the CosyVoice2 TTS system with a CDCD model.
- Architecture: The new model, CDCD-TTS, uses a Diffusion Transformer (DiT) backbone adapted from F5-TTS, with a softmax layer to predict FSQ speech tokens.
- Tokenization: The system utilizes 8-dimensional FSQ embeddings (Base3), resulting in a vocabulary of $3^8 = 6,561$ speech tokens.
- Training: Models were trained on 65k hours of English speech data (LibriLight, GigaSpeech, Emilia).
- Baselines: Comparisons were made against:
  - FSQ-original: CDCD with standard FSQ embeddings.
  - FSQ-perturb: CDCD with slightly perturbed FSQ embeddings.
  - FSQ-permute: CDCD with permuted token IDs (breaking the autoencoder mapping).
  - RVQ: CDCD using Residual Vector Quantization (EnCodec) tokens.
  - LLM Counterparts: CosyVoice2, F5-TTS, and CosyVoice3.

3. Key Contributions

Theoretical Connection: The paper establishes a formal link between forward diffusion parameters, the Euclidean distance between generated samples, and the KL divergence between reverse diffusion path measures.
FSQ Optimality: It formulates and verifies (both theoretically and empirically) that FSQ codebooks possess geometric properties (specifically, maximal average nearest-neighbor distance) that make them optimally suited for CDCD models.
Novel TTS Model: The authors propose CDCD-TTS, the first CDCD-based text-to-speech model relying on FSQ speech tokens. This model demonstrates that non-autoregressive diffusion can effectively generate discrete speech tokens.

4. Results

Numerical Findings: Experiments on toy data confirmed that FSQ codebooks outperform random and perturbed codebooks in terms of average nearest-neighbor distance and log-KL divergence between ground truth and generated distributions.
TTS Performance:
- Quality: The FSQ-original CDCD-TTS model achieved superior performance compared to the FSQ-perturb and FSQ-permute variants across Word Error Rate (WER), Speaker Similarity (SIM), and Mean Opinion Score (MOS). Notably, the FSQ-original model outperformed the LLM-based CosyVoice2 in WER (2.00% vs 2.57%) and MOS (4.119 vs 4.077).
- Efficiency: The CDCD-TTS backbone is approximately 10x smaller (45M parameters vs 0.5B LLM) and 5–10x faster than its LLM-based counterpart.
- Real-time Factor: The system achieves a real-time factor of 0.2–0.3, synthesizing speech 3–5x faster than real-time on a single GPU with 25 reverse diffusion steps.
- Zero-Shot Capabilities: The model maintains strong zero-shot voice cloning and emotion copying capabilities, outperforming the LLM-based baseline in speech intelligibility and naturalness.

5. Significance and Claims

The paper claims that the geometric structure of the latent space is a critical, often overlooked factor in the success of CDCD models. By demonstrating that FSQ tokenization provides a latent space structure that maximizes the separability of diffusion paths, the authors argue that FSQ is not merely a quantization technique but a theoretically optimal choice for continuous diffusion on categorical data.

The significance of this work lies in:

Challenging the Dominance of Autoregressive Models: It provides a viable, high-quality alternative to autoregressive LLMs for discrete data generation, offering significant gains in inference speed and model size.
Stabilizing CDCD Training: It offers a principled approach to designing latent spaces that mitigate the training instability historically associated with CDCD frameworks.
Practical Application: The successful deployment in TTS demonstrates that CDCD models can match or exceed the performance of state-of-the-art LLM-based systems while being significantly more efficient, suggesting a promising direction for future research in generative modeling for discrete modalities.

The authors remain modest regarding the "Best Accuracy Hypothesis," noting that the theoretical proof is currently limited to cases with equal prior token probabilities, and they identify handling non-uniform priors as a direction for future research.