Technical Summary: NüshuVoice: Reviving the Voice of Endangered Nüshu with Pitch-Aware Text-to-Speech

Problem Statement

Nüshu is an endangered phonetic script historically used by women in Jiangyong County, Hunan, China. While recent computational efforts have focused on the textual digitization, encoding, and visual recognition (OCR) of Nüshu, its acoustic modality remains largely unexplored. This gap is critical because Nüshu is a syllabic script where each character corresponds to a specific syllable, including initial, final, and tone, in the local Jiangyong dialect.

Building a Text-to-Speech (TTS) system for Nüshu presents extreme challenges due to severe data scarcity. Unlike modern low-resource settings that may possess several hours of continuous speech, available Nüshu recordings are archival, non-studio quality, and consist almost entirely of isolated syllable-level pronunciations rather than natural sentence-level utterances. Conventional TTS architectures, which rely on large paired text-speech datasets or external alignment supervision, are prone to unstable text-speech alignment and weak prosodic modeling under these conditions. Furthermore, the absence of Nüshu Automatic Speech Recognition (ASR) systems precludes standard objective evaluation metrics like Word Error Rate (WER).

Methodology

1. Dataset Construction: NüshuVoice

The authors constructed NüshuVoice, the first sentence-level multimodal TTS benchmark for Nüshu. The construction pipeline involved three rigorous stages:

• Data Collection: Textual data was sourced from The Mystery of Jiangyong Nüshu (approx. 63,665 characters), providing original glyphs, phonetic transcriptions (using IPA and a five-level pitch scale), and standard Chinese translations. Audio data was extracted from the Nüshu Pronunciation Electronic Dictionary, containing 873 isolated syllable recordings.
• Orthographic Filtering: A strict visual and semantic cross-referencing protocol was applied to map handwritten/printed glyphs to standardized Unicode forms. Sentences containing any character without a verified Unicode equivalent or structural discrepancy were discarded to ensure orthographic fidelity.
• Sentence-Level Audio Assembly: Since continuous recordings were unavailable, the authors developed a deterministic pipeline to assemble sentence-level audio. Verified Unicode characters were mapped to their corresponding isolated WAV files from the dictionary. These segments were temporally concatenated using Audacity to reconstruct continuous utterances, creating a one-to-one alignment between standardized text, IPA, and audio.
• Validation: Expert validation by two Nüshu domain experts confirmed a 93.6% correctness rate for the final dataset (1,252 utterances, ~98.8 minutes).

2. Model Architecture: Nüshu-PitchVITS

To address the extreme low-resource setting, the authors proposed Nüshu-PitchVITS, an extension of the Variational Inference with Adversarial Learning for End-to-End Text-to-Speech (VITS) framework.

• F0-Conditioned Framework: Recognizing that Nüshu annotations explicitly include a five-level pitch notation, the model incorporates an explicit fundamental frequency (F0) predictor as a prosodic inductive bias.
• Architecture Details:
  • Input: Tone-aware phonetic transcriptions (syllable + 5-level tone value).
  • Pitch Branch: A lightweight two-layer 1D convolutional network predicts a frame-level log-F0 sequence ($\hat{f}$) from the text encoder's hidden states.
  • Latent Injection: The predicted pitch representation is projected to the hidden dimension and added to the latent acoustic representation ($Z' = Z + E_{F0}$) before waveform decoding.
  • Alignment: The pitch predictor is excluded from the Monotonic Alignment Search (MAS) to avoid destabilizing alignment estimation in low-data regimes. Alignment is computed solely from prior statistics and flow-transformed posterior variables.
• Training Strategy: A two-stage training approach was employed, initialized from a pre-trained English VITS model:
  1. Stage 1: The waveform decoder and posterior encoder are frozen. Only the text encoder, pitch predictor, duration predictor, and normalizing flow are trained using KL-divergence, duration, and pitch prediction losses.
  2. Stage 2: All modules are unfrozen for end-to-end joint training using a combined loss function including adversarial, feature matching, reconstruction, and pitch losses.

Key Contributions

1. NüshuVoice Dataset: The creation of the first sentence-level Nüshu TTS dataset, aligning standardized Unicode text, IPA transcriptions, Chinese translations, and assembled archival audio.
2. Nüshu-PitchVITS: A novel TTS framework that leverages Nüshu's explicit five-level pitch notation as a prosodic inductive bias to stabilize synthesis in extreme low-resource conditions.
3. Systematic Evaluation: The first comprehensive objective and subjective evaluation of neural TTS for Nüshu, establishing new baselines and metrics for endangered phonetic scripts.
4. Cultural Preservation: A computational approach to making the pronunciation of Nüshu accessible, supporting the acoustic revitalization of this endangered writing system.

Experimental Results

The proposed method was evaluated against five strong baselines (Tacotron 2, FastSpeech 2, Glow-TTS, F5-TTS, and standard VITS) on the NüshuVoice test set.

• Objective Metrics:
  • Spectral Fidelity: Nüshu-PitchVITS achieved a Mel-Cepstral Distortion (MCD) of 3.11, significantly outperforming the strongest baseline (VITS at 5.19) and showing massive improvements over autoregressive models (which suffered MCD > 37).
  • Pitch Reconstruction: The model reduced F0 Root Mean Square Error (RMSE) to 13.23 (vs. 23.42 for VITS) and achieved a Pearson correlation of 0.92 (vs. 0.73 for VITS), demonstrating superior tonal accuracy.
• Subjective Metrics (MOS):
  • Intelligibility: Nüshu-PitchVITS achieved a near-perfect Intelligibility MOS of 4.97, whereas standard baselines scored near 1.0 (unintelligible).
  • Naturalness: The model achieved a Naturalness MOS of 3.31, comparable to ground-truth recordings (3.34).
• Spectral Analysis: Spectrograms indicated that while standard VITS exhibited over-smoothing and weakened harmonic contrast, Nüshu-PitchVITS produced sharper harmonic bands and more consistent pitch-related structures, confirming the efficacy of explicit F0 conditioning.

Significance and Claims

The paper claims that NüshuVoice and Nüshu-PitchVITS represent the first dedicated study on Nüshu TTS synthesis. The authors argue that for endangered phonetic scripts where acoustic data is scarce but tonal annotations are available, linguistically grounded prosodic modeling is critical.

The significance of this work lies in:

1. Acoustic Revitalization: Moving beyond visual recognition to recover the spoken realization and tonal dynamics of Nüshu, which is essential for preserving its oral and cultural heritage.
2. Methodological Insight: Demonstrating that explicit pitch conditioning can stabilize VITS architectures in extreme low-resource settings where conventional alignment mechanisms fail.
3. Resource Availability: Providing a public dataset and code to serve as a foundation for future research in low-resource TTS, endangered language revitalization, and the computational preservation of intangible cultural heritage.

The authors maintain a modest stance, acknowledging limitations such as the assembled nature of the audio (lacking natural coarticulation and discourse prosody), the limited speaker diversity due to archival constraints, and the exclusion of historical glyph variants without exact Unicode equivalents. They emphasize that the work aims to support preservation and documentation rather than replace community knowledge.