HybridCodec: Modeling Discrete and Continuous Representations for Efficient Speech Language Models

Technical Summary: HybridCodec

Problem Statement
Discrete audio representations have become the standard for integrating audio capabilities into Large Language Models (LLMs) and building multimodal systems. However, existing literature highlights a fundamental trade-off: while discrete tokens facilitate stable convergence and seamless LLM integration, the quantization process inevitably discards fine-grained acoustic details. This "quantization penalty" leads to performance degradation in downstream tasks, particularly the loss of speaker identity, micro-prosody, and acoustic richness, especially at low bit-rates. Current solutions often rely on task-specific adaptations (e.g., diffusion or masked modeling) that sacrifice the unified, generalizable framework provided by discrete LLMs.

Methodology
The authors propose HybridCodec, a novel framework that bridges the gap between discrete efficiency and continuous acoustic fidelity. The approach consists of two primary components:

1. HybridCodec (The Encoder/Decoder):

   • Architecture: An extension of the FocalCodec architecture, it employs a dual-path encoding strategy.
   • Discrete Path: Extracts temporally compressed discrete tokens ($z_q$) from the base audio features using Binary Spherical Quantization (BSQ).
   • Continuous Path: Computes the residual error ($x_{res}$) between the original features and the quantized approximation. This residual is compressed by a dedicated focal encoder into a dimensionality-reduced continuous representation ($\bar{x}_{res}$).
   • Decoding: The decoder reconstructs the signal by upsampling the discrete tokens and the continuous residuals separately, then summing them ($\hat{x}_{base} = \hat{x}_{quant} + \hat{x}_{res}$) before passing them to a Vocos decoder for waveform synthesis.
   • Temporal Resolution: The system supports variable frame rates (50 Hz down to 6.25 Hz) by adjusting the downsampling strides of the residual encoder.

2. HybridLM (The Language Model):

   • Architecture: A decoder-only Transformer designed to process the hybrid representations.
   • Unified Decoding: It unifies two operational modes within a single network using Adaptive Layer Normalization (AdaLN):
     • Autoregressive (AR) Phase: Generates discrete tokens to establish semantic structure.
     • Non-Autoregressive (NAR) Phase: Predicts continuous residuals in a single forward pass to recover high-fidelity acoustic details.
   • Inference Efficiency: By generating discrete tokens at a lower frame rate (e.g., 12.5 Hz) and refining them with a single NAR step, the model drastically reduces the total number of autoregressive steps required compared to full-rate discrete generation.

Key Contributions

1. HybridCodec Framework: A neural audio codec that jointly extracts time-reduced discrete tokens and models remaining information as dimensionality-reduced continuous residuals, effectively mitigating the rate-distortion trade-off.
2. HybridLM Architecture: A unified Transformer decoder that leverages AdaLN to multiplex AR token generation and NAR residual prediction, allowing for mixed discrete-continuous prompts at different temporal scales.
3. Unified Downstream Performance: A single framework capable of handling major speech tasks (Automatic Speech Recognition and Text-to-Speech) while restoring fine-grained information lost in purely discrete models.

Experimental Results
Evaluated on the LibriTTS dataset, the approach demonstrates significant improvements over discrete-only baselines, particularly at ultra-low frame rates:

• Resynthesis: At 12.5 Hz, HybridCodec achieved a differential Word Error Rate (dWER) of 1.47, a substantial improvement over the 7.94 dWER of the discrete-only FocalCodec baseline. Speaker similarity (SpkSim) remained high (97.1) even at 12.5 Hz.
• Text-to-Speech (TTS): In zero-shot settings at 12.5 Hz, the hybrid method more than doubled the UTMOS score (4.10 vs. 1.99) and reduced dWER by more than half (14.79 vs. 32.97) compared to discrete-only models. At an extreme 6.25 Hz, the hybrid approach maintained a UTMOS of 3.08, vastly outperforming the discrete baseline (1.44).
• Automatic Speech Recognition (ASR): The inclusion of continuous residuals consistently improved transcription accuracy. At 50 Hz, the Word Error Rate (WER) dropped from 28.11 to 23.36. This improvement persisted even at high compression levels (12.5 Hz and 6.25 Hz), confirming that continuous acoustic cues enhance rather than interfere with semantic representations.

Significance
The paper claims that HybridCodec represents the first approach to unify discrete and continuous refinement within a single Transformer architecture. By leveraging the efficiency of discrete tokens for semantic structure and continuous residuals for acoustic fidelity, the framework achieves high-fidelity speech synthesis at ultra-low frame rates (down to 6.25 Hz). This significantly reduces the computational cost of inference (fewer autoregressive steps) while restoring the speaker characteristics and prosodic nuances that are typically lost in purely discrete representations. The work suggests a viable path toward efficient, high-quality, end-to-end multimodal speech systems.