Technical Summary: HoliTok

Problem Statement

Unified speech foundation models aim to perform both speech understanding (e.g., recognition) and generation (e.g., synthesis) within a shared parameter space. However, existing speech tokenization methods fail to simultaneously satisfy the conflicting requirements of being learnable by language models, decodable into high-fidelity waveforms, and informative for semantic understanding.

• Traditional acoustic features (e.g., mel-spectrograms) preserve signal structure but produce dense, redundant sequences that are difficult for downstream models to learn.
• Self-supervised representations offer rich semantic information but are often not naturally decodable into high-quality waveforms.
• Discrete tokenizers (codec-based) compress speech into language-friendly symbols but introduce quantization artifacts and modeling complexity.
• Existing continuous tokenizers often optimize for reconstruction or synthesis in isolation, lacking a shared representation space that supports unified generation-understanding architectures without complex, task-specific architectural adjustments.

Consequently, the burden of unification is shifted from the representation itself to increasingly complex model designs, such as multi-stream tokenizers or decoupled modules.

Methodology

The authors propose HoliTok, a continuous Holistic speech Tokenization model designed to serve as a unified interface for generation and understanding.

Architecture

HoliTok is built on a low-latency Variational Autoencoder (VAE) backbone:

• Encoder: A causal convolutional network with a 2-frame lookahead, downsampling 48 kHz audio to a 25 Hz sequence of 128-dimensional continuous latents.
• Temporal Variational Bottleneck: An LSTM-based bottleneck that introduces stochasticity via a reparameterization trick, regularized by a normalizing flow to enhance expressiveness.
• Decoder: A BigVGAN-style generator that reconstructs the 48 kHz waveform from the latent sequence.
• Supervision Network: A downstream-aware component (0.6B Transformer encoder + Qwen2.5-0.5B decoder) used to inject semantic and task-specific information during training.

Progressive Training Strategy

HoliTok employs a three-stage progressive training recipe to balance fidelity, learnability, and semantic richness:

1. Stage I (High-Fidelity Autoencoding): A deterministic autoencoder is trained to establish a reliable reconstruction manifold using multi-scale spectral, adversarial, and feature-matching losses. This ensures the decoder can faithfully reconstruct waveforms before introducing stochasticity.
2. Stage II (Implicit Fidelity Transfer): The encoder and decoder are frozen. Only the temporal variational bottleneck is trained with a weak KL regularization. This transfers the deterministic space into a stochastic one while keeping latent samples close to the high-fidelity decoding region, preventing the loss of acoustic details.
3. Stage III (Downstream-Aware Enrichment): The full model (encoder and bottleneck) is jointly optimized with a strong KL weight. This stage enriches the latent space through:
   • Multi-granularity Distillation: Aligning latents with frozen teacher representations (WavLM for frame-level context, x-vector for speaker identity).
   • Multi-task Supervision: Using a language-modeling objective to predict task targets (ASR, emotion, captioning, etc.) conditioned on the latent representation. This forces the latent space to retain information critical for understanding, even if not strictly necessary for waveform reconstruction.

Downstream Unified Modeling

To evaluate the tokenizer, the authors construct a unified AR+DiT (Autoregressive + Diffusion Transformer) model:

• Understanding: An LLM processes mixed text and audio latent patches to predict the next text token (ASR).
• Generation: The LLM predicts semantic hidden states, which condition a DiT-based flow-matching head to predict the next latent patch (TTS).
• Evaluation: The system tests whether the same latent sequence can robustly support both tasks without additional architectural tricks.

Key Contributions

1. HoliTok Tokenizer: A continuous speech tokenizer that encodes 48 kHz speech into a compact 25 Hz, 128-dimensional latent sequence.
2. Progressive Training Recipe: A novel three-stage strategy that sequentially establishes fidelity, introduces variational regularity, and enriches semantics, ensuring the latent space remains both decodable and learnable.
3. Unified Interface: Demonstration that a single continuous representation can effectively serve as the shared interface for an AR+DiT architecture, supporting both high-quality synthesis and accurate recognition without task-specific encoders or multiple token streams.

Experimental Results

The authors evaluate HoliTok on reconstruction, speech synthesis, and unified modeling tasks against baselines like Semantic-VAE and MingTok-Audio.

• Reconstruction: HoliTok achieves competitive signal fidelity (PESQ, STOI) with a high compression ratio (7.5×) and low latency (25 TPS). It outperforms baselines in preserving speaker and emotion similarity (SPKSIM, EMOSIM).
• Speech Synthesis: In zero-shot and controllable TTS settings, HoliTok demonstrates high learnability, achieving the best win rates in expressive and paralinguistic dimensions and strong performance in controllable synthesis (low WER, high CLSP scores).
• Unified Generation-Understanding:
  • HoliTok-Base (using only the VAE latents) already outperforms baselines in TTS intelligibility while maintaining comparable ASR performance.
  • HoliTok-Unite (incorporating the causal semantic encoder from Stage III) further improves both TTS and ASR, significantly reducing WER in both tasks compared to HoliTok-Base and baselines.
  • In contrast, baselines like Semantic-VAE degrade sharply in TTS when used in a unified setting, and MingTok-Audio shows an imbalance favoring understanding over generation.

Significance and Claims

The paper claims that HoliTok serves as an effective speech tokenizer and a foundational representation interface for unified spoken language modeling.

• Robustness: Unlike existing representations that require complex architectural modifications or incremental training designs to unify tasks, HoliTok operates robustly in a unified AR+DiT architecture without additional optimization tricks.
• Bridging the Gap: It successfully bridges the gap between semantic abstraction, acoustic fidelity, and model learnability, suggesting that a single continuous space can satisfy the joint requirements of generation (acoustic preservation) and understanding (semantic accessibility).
• Modest Scope: The authors acknowledge that their current evaluation focuses on speech-centered tasks (reconstruction, TTS, ASR). They do not claim generalization to broader audio domains (e.g., music, environmental sound) or other generative architectures (e.g., pure DiT) within this work, noting these as directions for future research.

The code is available at: https://github.com/bovod-sjtu/HoliTok.