Technical Summary: FormalASR

1. Problem Statement

Automatic Speech Recognition (ASR) systems are traditionally optimized for verbatim transcription, faithfully reproducing the spoken surface form. While effective for benchmarks measuring Character Error Rate (CER), this output often includes disfluencies (e.g., "um," "uh"), false starts, repetitions, and informal sentence structures. These characteristics render the output unsuitable for downstream writing-oriented applications such as meeting minutes generation, document dictation, or voice-controlled editors, which require formal, well-formed text.

Current workarounds typically involve a two-stage pipeline: an ASR model generates a verbatim transcript, followed by a separate Large Language Model (LLM) to rewrite the text into a formal style. However, this approach introduces significant latency, doubles memory footprint, and complicates on-device deployment due to the requirement of running two models simultaneously. Conversely, large multimodal models capable of single-pass formal transcription often rely on cloud APIs, incurring costs and raising privacy concerns that preclude edge deployment.

2. Methodology

The authors propose FormalASR, an end-to-end approach that directly maps spoken Chinese audio to formal written text using a single compact model, eliminating the need for auxiliary LLMs at inference time.

2.1. Dataset Construction

To enable supervised training for this task, the authors constructed two large-scale datasets by rewriting existing verbatim corpora:

• WenetSpeech-Formal: Derived from the WenetSpeech corpus, containing 969,201 training samples.
• Speechio-Formal: Derived from the Speechio benchmark, containing 43,178 test samples across 27 domain-specific subsets (lectures, interviews, meetings, etc.).

Construction Pipeline:

1. Input: Original audio and verbatim transcriptions.
2. LLM Rewriting: The DeepSeek-V3.2 model was prompted to rewrite verbatim transcripts into formal written Chinese. This process involved removing filler words, restructuring sentences to follow written conventions, normalizing punctuation, and correcting recognition errors while preserving semantic content.
3. Quality Filtering: Automatic filtering discarded low-quality rewrites based on embedding similarity (semantic consistency), edit distance (ensuring meaningful formalization without hallucination), and the presence of obvious artifacts.

2.2. Model Architecture and Training

The authors fine-tuned Qwen3-ASR (an audio-language model with a Whisper-style encoder and an autoregressive Qwen decoder) at two scales: 0.6B and 1.7B parameters.

• Training Objective: Standard supervised fine-tuning (SFT) using maximum likelihood estimation. The model learns to jointly perform acoustic-to-text alignment, disfluency removal, and style transfer within a single conditional generation process.
• Inference: Decoding is performed directly on audio input to produce formal text, maintaining system complexity identical to standard single-model ASR deployments.

3. Key Contributions

1. Datasets: The open-sourcing of WenetSpeech-Formal and Speechio-Formal, the first large-scale Chinese spoken-to-formal datasets constructed via LLM-based rewriting and quality filtering. These enable end-to-end training without auxiliary models at inference.
2. Models: The release of FormalASR-0.6B and FormalASR-1.7B, compact audio-language models fine-tuned for direct spoken-to-formal transcription.
3. Performance: Demonstration that compact models can achieve significant improvements in formalization quality over verbatim baselines while remaining suitable for on-device deployment.

4. Experimental Results

Experiments were conducted on both in-domain (WenetSpeech-Formal test set) and cross-domain (Speechio-Formal test set) benchmarks.

4.1. Accuracy Metrics

FormalASR consistently outperformed verbatim baselines (Qwen3-ASR and Whisper large-v3) across all metrics:

• Character Error Rate (CER): FormalASR achieved up to a 37.4% relative CER reduction over verbatim baselines on the cross-domain Speechio-Formal benchmark (e.g., 1.7B model reduced CER from 0.2393 to 0.1499).
• ROUGE-L and BERTScore: Both metrics improved, confirming that the model preserves semantic content while removing disfluencies. For instance, on WenetSpeech-Formal, the 1.7B model improved ROUGE-L from 0.8571 to 0.8896 and BERTScore from 0.9268 to 0.9439.
• Scale Effect: The 1.7B model consistently outperformed the 0.6B model, suggesting larger decoders provide stronger capacity for linguistic formalization.

4.2. Inference Efficiency

• Token Reduction: By removing filler words and disfluencies, FormalASR produces shorter output sequences. On average, FormalASR-1.7B reduced output tokens by 22.8% on WenetSpeech-Formal and 14.3% on Speechio-Formal compared to the verbatim baseline.
• Latency: The reduction in sequence length led to lower decoding latency. The latency advantage grew super-linearly with sentence length; for long utterances (40–49 tokens), FormalASR-1.7B reduced latency by approximately 388 ms compared to the verbatim baseline.

4.3. Quantization and On-Device Feasibility

The authors evaluated GGUF quantization to validate on-device potential:

• 8-bit (Q8 0): Near-lossless compression. Model size reduced by ~47% (e.g., 1.7B model from 3.80 GB to 2.03 GB) with negligible quality degradation (CER increase of only 0.0001).
• 4-bit (Q4 K): Strong compression (~72% size reduction) with moderate quality trade-offs. Notably, the 1.08 GB 4-bit 1.7B model still outperformed the 1.46 GB 0.6B model in full precision, indicating that model capacity, rather than numerical precision, is the primary determinant of formalization quality.

5. Significance and Claims

The paper claims that FormalASR is the first work to fine-tune a compact audio-language model end-to-end for spoken-to-formal Chinese transcription. Its primary significance lies in demonstrating that modern ASR models possess the latent capacity for linguistic formalization; they simply require appropriate supervision to activate this capability without increasing model size or inference-time complexity.

By achieving high-quality formalization in a single forward pass, FormalASR addresses the critical gap between verbatim ASR output and the formal text required by downstream applications. It offers a lightweight, on-device solution that avoids the latency, memory costs, and privacy concerns associated with two-stage pipelines or cloud-based multimodal models. The authors conclude that their approach enables practical deployment for real-time transcription in privacy-sensitive or resource-constrained environments.