Technical Summary: Stable Autoregressive Speech Generation with Low-Frame-Rate High-Dimensional Continuous Tokens

1. Problem Statement

The paper addresses the fundamental tension in autoregressive (AR) speech and audio generation between representational capacity and long-horizon stability.

• The Trade-off: High-bandwidth, high-dimensional representations preserve signal detail and support high-fidelity reconstruction but are difficult to predict. In AR systems, small local prediction errors accumulate over time, leading to distribution drift (e.g., in loudness, timbre, or spectral quality) and eventual generation collapse. Conversely, low-bitrate or highly compressed representations simplify AR modeling but discard critical information, capping reconstruction fidelity.
• The Gap: Existing approaches often rely on externally pretrained Self-Supervised Learning (SSL) or Automatic Speech Recognition (ASR) models to define semantic spaces, or they use discrete token hierarchies (e.g., RVQ) that increase model complexity. The authors ask whether a low-frame-rate, high-dimensional, continuous token formulation can be co-designed with a streaming generation framework to achieve robust high-fidelity reconstruction, strong single-token predictability, and stable long-form synthesis without relying on external pretrained models or post-training stages.

2. Methodology

The proposed solution consists of two coupled components: Locodec, a tokenizer that shapes the latent space geometry, and MP-ELD, a generative framework designed to mitigate error accumulation.

2.1 Locodec: Locally Encoded Codec

Locodec is designed to create a high-dimensional spherical token space that is organized around a lower-dimensional, interpolatable core manifold while maintaining an energy hierarchy for identifiability.

• Spherical Token Space: Tokens are constrained to a high-dimensional sphere ($S^{D-1}$). This removes the radial degree of freedom from the prediction task, focusing the generative model on angular discrepancies.
• Low-Dimensional Core Manifold: A learnable orthogonal projection maps high-dimensional tokens to a lower-dimensional core manifold ($d_{core} \ll D$). A bidirectional commitment loss aligns the high-dimensional tokens with this core. The core manifold is subjected to strong spherical corruption (random rotations) during training, forcing it to encode the most reconstruction-critical information and encouraging strong interpolatability.
• Postfix Dimension Dropout (PDD): To induce identifiability, the tokenizer employs a postfix dimension dropout mechanism. During training, dimensions are dropped from the end of the token vector with a specific probability. This creates an availability bias where lower-indexed dimensions are retained more frequently. Consequently, the model learns to allocate higher energy to these "prefix" dimensions, creating an energy hierarchy that makes core content more robust to corruption and easier to predict.
• Architecture: The tokenizer uses a fully local encoder (no cross-token interactions) based on MDCT coefficients and a stacked causal ConvNeXt1D decoder. It supports both monaural and stereo audio via a mid/side formulation.

2.2 MP-ELD: Multi-Path Encoder-LM-Decoder

MP-ELD is a single-token AR flow-matching framework designed to mitigate error accumulation caused by inconsistent guidance under Classifier-Free Guidance (CFG).

• Multi-Path Information Routing: Instead of a single conditioning signal, MP-ELD constructs three distinct conditioning pathways:
  1. Local-Continuity ($c_{lc}$): Derived from the current token to enforce short-range acoustic continuity.
  2. Self-Consistency ($c_{sc}$): Derived from the token history to maintain long-horizon stability and internal attribute consistency (e.g., timbre).
  3. Alignment-Consistency ($c_{ac}$): Derived from the external control signal (e.g., text) to align generation with the prompt.
• Orthogonal Residual Conditioning: These paths are orthogonalized via a Gram-Schmidt transform to ensure they encode residual information.
• Multi-Path Residual CFG: During inference, the model runs three separate forward passes (or combinations thereof) to generate velocity fields for each path. The final velocity is a linear combination of residuals:
  $$v_\tau = v_L + \lambda_{sc}(v_{LS} - v_L) + \lambda_{ac}(v_{LSA} - v_{LS})$$
  This allows independent control over acoustic consistency ($\lambda_{sc}$) and content alignment ($\lambda_{ac}$).
• Time-Dependent Guidance: To prevent early-time instability, the self-consistency guidance weight $\lambda_{sc}$ is scheduled to increase gradually over the bridge time $\tau$, avoiding aggressive extrapolation when the signal is highly noisy.
• Direction-Only Modeling with Flexible Magnitude: The model predicts a spherical endpoint to define a tangent direction. Crucially, the magnitude of the combined velocity is NOT re-normalized to the fixed SFM limit after CFG. While the direction is derived from the endpoint, the magnitude is allowed to vary based on the CFG residuals. This flexibility permits the guided trajectory to have a path length different from the standard geodesic, accommodating curved paths that may be necessary to reach the desired endpoint under the learned vector field, rather than forcing a fixed step size that could lead to under-stepping or failure to reach the target.

3. Key Contributions

1. Locodec Tokenizer: A novel tokenizer that shapes a high-dimensional continuous token space without external SSL/ASR supervision. It combines spherical normalization, a low-dimensional core manifold constraint, and a postfix dimension dropout strategy to improve both interpolatability and identifiability.
2. MP-ELD Framework: A flow-matching generative model that explicitly disentangles information pathways (local, self-consistent, and aligned) and uses multi-path residual CFG to mitigate AR error accumulation and distribution drift.
3. Joint Design Principle: The paper demonstrates that balancing AR stability and reconstruction fidelity requires co-designing the representation space geometry and the generative framework, rather than relying solely on larger models or external priors.
4. Empirical Findings on Token Frame Rate: The study suggests that low frame rates (e.g., 8 Hz) act as a "semantic-acoustic resolution knob," naturally aggregating content over longer spans (benefiting WER and stability) but potentially reducing sensitivity to fine-grained acoustic details (impacting Speaker Similarity).

4. Experimental Results

Experiments were conducted on the Seed-TTS-eval dataset (Chinese and English) and internal long-form test sets. The models were trained from scratch without pretrained text LMs, SSL, or ASR.

• Reconstruction Quality: Imposing a low-dimensional core manifold and PDD did not degrade full-dimensional reconstruction quality. PDD successfully induced a coordinate-wise energy hierarchy, where prefix dimensions carried significantly more energy than postfix dimensions.
• Predictability: Generative models trained on tokens with a moderate core dimension ($d_{core}=32$) and PDD achieved the lowest training loss and fastest convergence, indicating these tokens are the most predictable.
• Generation Performance (Short-Form): On Seed-TTS-eval, the best configuration (32/✓) achieved a Word Error Rate (WER) of 0.95% (Chinese) and 1.87% (English), competitive with state-of-the-art AR systems.
• Long-Form Stability:
  • CFG Sensitivity: The study revealed that alignment-consistency guidance ($\lambda_{ac}$) primarily controls content alignment (WER), while self-consistency guidance ($\lambda_{sc}$) controls acoustic stability (Speaker Similarity).
  • Drift Mitigation: Time-dependent scheduling of $\lambda_{sc}$ was crucial. Constant high guidance led to severe drift in long-form synthesis (e.g., segment-level Speaker Similarity dropping from 0.74 to 0.40). Delayed schedules ($\gamma > 0$) significantly improved long-horizon stability.
  • PDD Impact: PDD consistently improved long-form stability. For example, switching from 32/× to 32/✓ reduced long-form WER from 20.80% to 9.73% and improved the 5th-segment Speaker Similarity from 0.419 to 0.578.
• Comparison: The proposed system achieves competitive WER without external models. However, it exhibits a lower Speaker Similarity (SIM) compared to systems using explicit semantic-acoustic factorization or higher frame rates, supporting the hypothesis that low-frame-rate tokens favor content aggregation over fine-grained acoustic detail.

5. Significance and Claims

The paper claims that low-frame-rate, high-dimensional continuous tokens are a viable target for stable AR speech generation, provided the representation space and generative framework are jointly optimized.

• Modest Claims: The authors explicitly state they do not claim a universal scaling law or that their design dominates data/model scale. Instead, they isolate the effects of token-space shaping and information routing.
• Key Insight: The work challenges the necessity of external SSL/ASR supervision for shaping token spaces, showing that a reconstruction-first tokenizer with geometric constraints can achieve similar stability benefits.
• Limitations: The authors acknowledge that while their system achieves strong content alignment (low WER) and stability, it currently lags behind the strongest prior systems in Speaker Similarity. They attribute this to the "semantic-acoustic trade-off" inherent in low-frame-rate tokens, which aggregate content effectively but may smooth over fine-grained acoustic variations.
• Future Directions: The paper suggests that future work could explore introducing semantic-acoustic factorization into the low-rate continuous space without external supervision, scaling the decoder components, and developing dynamic CFG schedules that adapt to generation length.