CN-NewsTTS Bench: a target-level automatic benchmark for raw-input Chinese news TTS pronunciation

Technical Summary: CN-NEWSTTS BENCH

Problem Statement
Chinese news text frequently contains dense, non-standard written forms—such as sports scores (e.g., 96-91), hyphenated model names (e.g., 苏 -27), ranges, unit symbols, percentages, and mixed Chinese-Latin-digit names. While unambiguous to human editors, these strings present significant challenges for Text-to-Speech (TTS) systems. A TTS model may preserve the written string while altering the spoken meaning (e.g., reading a score as a range, or a military model as a negative number). Existing evaluation methods, such as subjective Mean Opinion Score (MOS) tests or generic ASR round-trip comparisons, are too coarse to track these specific, high-impact reading decisions. Furthermore, current benchmarks often rely on user-side normalization, Large Language Model (LLM) rewriting, or SSML hints, failing to evaluate the "raw-input" behavior of deployed TTS APIs.

Methodology
The paper introduces CN-NewsTTS Bench v0.1, an open, target-level automatic benchmark designed to evaluate Chinese news TTS pronunciation from raw text without external rules or manual edits.

• Data Construction: The dataset consists of 1,000 deterministic, synthetic news-style sentences generated from category-specific templates and lexicons (covering sports, military models, technical units, etc.). It includes a 200-record development set and an 800-record public test set, containing 992 auto-evaluable targets.

• Evaluation Protocol (Raw Input Product Track): Systems are evaluated on their ability to process original Chinese news text directly. Provider-internal normalization is permitted, but external rule frontends, LLM rewrites, SSML hints, and manual text edits are strictly excluded. A fixed voice is used for all samples to isolate pronunciation performance.

• Three-ASR Scoring Protocol: To avoid the subjectivity of human listening and the limitations of single ASR models, the benchmark employs a heterogeneous ensemble of three ASR routes:

  1. MiMo API ASR (API-based)
  2. SenseVoiceSmall (Open-source local)
  3. Paraformer-zh (Open-source local)

  The scorer normalizes transcripts (Unicode, case, punctuation) and matches them against predefined "positive" (correct) and "negative" (incorrect) reading patterns. A target is marked correct if a positive pattern is found, wrong if a negative pattern is found, and unknown otherwise. Final results use majority voting (at least two routes must agree).

• Metrics: The primary metric is Strict Auto Accuracy (correct targets divided by total auto-evaluable targets). The authors also report Coverage (targets resolved as correct or wrong) and Resolved Accuracy (correct targets divided by resolved targets) to prevent systems from appearing accurate simply by failing to resolve difficult targets.

Key Contributions

1. Open Deterministic Split: A fixed dev/public split for Chinese news pronunciation targets, ensuring reproducibility.
2. Raw Input Product Track: A specific evaluation track that excludes user-side normalization, testing the end-to-end behavior of deployed TTS APIs.
3. Target-Level Schema: A granular evaluation schema distinguishing between positive and negative readings for specific targets.
4. Three-ASR Automatic Scoring: A robust protocol using an ensemble of ASR models with route diagnostics and ablation studies to handle ambiguity.
5. Reproducible Leaderboard: Initial results for seven product TTS systems, providing a baseline for future comparisons.

Results
The benchmark evaluated seven major TTS systems: Volcano/Doubao, Azure, Google, MiniMax, Aliyun, MiMo, and AWS Polly.

• Performance Variance: There is substantial variation in performance. The best system (Volcano) achieved a strict accuracy of 0.879, while several widely used systems scored below 0.60.
• Category Difficulty: Performance varies significantly by category. Systems largely solved percentages, vehicle models, and abbreviations. However, sports scores were the most difficult category (strict accuracy as low as 0.000 for some systems), often misread as ranges or subtractions. Unit symbols had the highest "unknown" rate due to ASR limitations in exposing symbol-level readings.
• ASR Diagnostics: The three-route ensemble showed that while individual routes had similar strict accuracy, their coverage differed. MiMo API ASR was conservative (high resolved accuracy but many unknowns), while local models resolved more targets. The majority vote reduced the impact of individual route errors.
• Failure Modes: For systems scoring zero on sports scores, the dominant failure was reading score hyphens as ranges (e.g., interpreting "96-91" as "96 to 91" rather than "96 vs 91").

Significance and Claims
The paper claims that CN-NewsTTS Bench v0.1 makes a common production failure mode—incorrect pronunciation of compact Chinese news forms—measurable and reproducible. By fixing the data split, ASR transcripts, scorer, and category diagnostics, the benchmark provides a standardized way to evaluate raw-input TTS behavior. The results indicate that despite advances in TTS, reading these specific non-standard forms correctly remains a practical challenge for current commercial products. The authors emphasize that the benchmark is an automatic tool intended to complement, not replace, human listening tests, particularly for detecting tonal errors that ASR text may hide.