Technical Summary: SPARCLE

Problem Statement

Recent advances in text-to-speech (TTS) synthesis have shifted from phoneme-based representations to direct grapheme (character) modeling. While phonemes address the one-to-many mapping between text and acoustics, they rely on Grapheme-to-Phoneme (G2P) conversion systems. These G2P systems present significant limitations: they require accent- and dialect-aware models, depend on expensive phoneme labels or handcrafted pronunciation dictionaries, and often fail to scale or generalize across different speaker domains (e.g., North American vs. British English).

Conversely, while character-based models can outperform phoneme-based systems when sufficient training data is available, they struggle in low-resource settings due to pronunciation ambiguity. A single grapheme sequence (e.g., "read") can yield multiple acoustic realizations depending on context, tense, or speaker identity, a nuance that standard character embeddings fail to capture without explicit phonetic supervision.

Methodology

The authors propose SPARCLE (SPeaker-aware Aligned Representations via Contrastive Language Embeddings), a model designed to enrich grapheme inputs with fine-grained, context-specific acoustic information without requiring explicit phoneme annotations.

Core Architecture

SPARCLE functions as a character-level encoder that aligns graphemes with acoustic representations conditioned on speaker identity. The architecture consists of three main components:

1. Character Alignment and Acoustic Extraction:

   • The model leverages the Wav2Vec2 family of models to extract acoustic embeddings.
   • To achieve character-level alignment, transcripts are forced-aligned to audio, mapping each character to a specific set of Wav2Vec2 embedding vectors.
   • Since the mapping is inherently one-to-many (a single character corresponds to multiple acoustic frames), an attention pooling mechanism is employed. A learned projection of the character embeddings generates attention weights over the associated acoustic frames, producing a single, fixed-size pooled audio embedding per character.

2. Speaker Conditioning:

   • To capture speaker identity without learning a new embedding matrix (which would hinder generalization to unseen speakers), SPARCLE utilizes FaCodec timbre embeddings.
   • These embeddings are extracted from a pretrained, speaker-agnostic model and provide strong separation between speakers, even with accented speech.
   • The speaker embedding is unit-normalized and prepended to the character embeddings before processing. Random dropout ($p=0.5$) is applied to the speaker token to prevent overfitting to the training speaker set.

3. Character Transformer:

   • The core is a standard encoder-style transformer (12 blocks, 12 attention heads, 768 embedding dimension) operating on uppercase letters and spaces.
   • To explicitly model local context (e.g., co-articulation effects), a 1D convolution with a kernel size of 3 is applied over character embeddings. These short-context embeddings are concatenated with the original character embeddings and projected to the transformer's input dimension.

Training Objective

SPARCLE is trained using a contrastive learning objective. The goal is to align the character embeddings (augmented with local context and speaker conditioning) with their corresponding pooled acoustic embeddings from the frozen Wav2Vec2 model.

• Positive Pairs: A character embedding and its corresponding pooled acoustic embedding.
• Negative Pairs: All other character-acoustic pairs within the batch.
• The loss is computed using cosine similarity with a temperature parameter of 0.1.

Key Contributions and Experimental Setup

The paper evaluates SPARCLE as a drop-in replacement for standard character embeddings in two distinct TTS backends: ParrotTTS (a modular system predicting discrete units) and VITS (an end-to-end model).

• Pre-training: SPARCLE is pre-trained on the LibriSpeech-960h dataset (US English) using forced alignment to create character-acoustic pairs.
• Downstream Evaluation: The model is fine-tuned on VCTK v0.92 (predominantly British English), creating a domain shift scenario to test generalization.
• Low-Resource Regimes: Experiments are conducted across training budgets ranging from 10 minutes to 10 hours, simulating extreme low-resource conditions.
• Fine-tuning Strategies: The authors evaluate different adaptation depths:
  • Frozen: SPARCLE acts as a fixed feature extractor.
  • Partial Fine-tuning: Only the last $K$ transformer blocks are unfrozen ($K \in \{1, 3, 7\}$).
  • Fully Unfrozen: All SPARCLE layers are trained jointly with the TTS backend.

Results

The experiments demonstrate that SPARCLE significantly improves generation quality, particularly in low-resource settings and when speaker conditioning is applied.

1. Pronunciation Accuracy (WER):

   • SPARCLE consistently outperforms standard character embeddings across all data budgets.
   • In the 10-minute regime, SPARCLE reduces Word Error Rate (WER) from 85.7% (baseline) to 42.2% (with timbre conditioning and $K=7$).
   • In the 1-hour regime, WER drops from 24.7% to 7.5%.
   • SPARCLE also outperforms phoneme-based models (using standard G2P) in most low-resource settings, particularly highlighting the failure of North American-centric G2P dictionaries (like g2pE) on British English data (VCTK).

2. Speaker Consistency (EER):

   • SPARCLE improves Equal Error Rate (EER), indicating better speaker consistency.
   • The best results are achieved with partial fine-tuning ($K=7$) and timbre conditioning. For example, at 10 hours of data, EER improves from 1.17% (baseline) to 0.76%.

3. Adaptation Trade-offs:

   • Partial fine-tuning yields the best balance between adapting to the downstream domain (VCTK) and preserving the pre-training signal (LibriSpeech). Fully unfrozen models often degrade performance, suggesting overfitting or forgetting of the robust pre-trained representations.
   • Timbre conditioning is most effective when the model is allowed to adapt (partial fine-tuning). When SPARCLE is frozen, adding timbre conditioning can sometimes degrade performance.

4. Backend Generality:

   • While VITS showed high WER in extreme low-resource settings (121.7%), SPARCLE integration still yielded a substantial improvement in speaker consistency (EER dropping from 5.6% to 3.9%), suggesting that improved text representations benefit speaker consistency even when data scarcity limits overall intelligibility.

Significance and Claims

The paper claims that SPARCLE successfully encodes rich acoustic information into grapheme representations, enabling high-quality TTS even with minimal downstream training data (as little as 30 minutes across speakers).

Key claims regarding the significance of this work include:

• Elimination of G2P Dependency: SPARCLE removes the need for expensive, dialect-specific G2P systems and pronunciation dictionaries, which are bottlenecks for scaling and multilingual adaptation.
• Robustness to Domain Shift: The model demonstrates strong generalization capabilities, transferring acoustic knowledge from US English (LibriSpeech) to British English (VCTK) without explicit domain adaptation.
• Future Directions: The authors suggest that because SPARCLE requires only aligned audio-text pairs, it is readily extensible to multilingual speech. Furthermore, its ability to condition on speaker identity without explicit IDs opens pathways for zero-shot speech generation using reference audio or descriptors. Finally, the authors note that SPARCLE embeddings are well-suited for modern Speech Language Models (like VALL-E) that generate discrete speech tokens autoregressively, offering a learned alternative to traditional G2P modules.