Technical Summary: Transcription Policy as a Latent Variable

1. Problem Statement

Current large-scale Automatic Speech Recognition (ASR) systems treat the transcription policy—the decision to output verbatim text (including disfluencies, fillers, repetitions, and paralinguistic sounds) versus intended text (clean, fluent content)—as an uncontrolled latent variable. This ambiguity arises because models are typically trained on heterogeneously annotated data without explicit control signals.

The authors identify three primary consequences of this ambiguity:

1. Decoding Instability: On disfluent speech, beam search yields significantly different transcripts for the same audio (median character-level beam divergence ~15%) because the model lacks a signal to select between competing transcription strategies.
2. Evaluation Confounding: Standard Word Error Rate (WER) conflates content errors with style differences. The authors demonstrate that up to 60% of reported WER on conversational benchmarks (e.g., AMI) reflects style mismatch rather than actual recognition failure.
3. Ill-Defined Timing: If a model inconsistently decides whether to emit a word repetition or a filler, the cross-attention distribution for that temporal region lacks a stable target, rendering word-level timestamps conceptually undefined and practically imprecise.

2. Methodology

The authors propose that the core challenge is capability control rather than capability acquisition, arguing that pretrained models already encode both styles but require explicit activation. They introduce three complementary mechanisms to unambiguously activate these latent capabilities using the OpenAI Whisper-medium architecture as a base.

A. Explicit Style Control via Mode Tags

The authors introduce discrete decoder-prefix tokens (mode tags) to parameterize the transcription policy as a binary choice between verbatim and intended modes.

• Design: They extend the vocabulary with specific tokens: [verbatim 1..3] for disfluency control, [sound 1,2] for paralinguistic events, and [intended 1..5] for clean output.
• Coverage-Aware Partitioning: To handle heterogeneous annotation quality (e.g., corpora with disfluency labels but no sound labels), the tags are partitioned. During training, samples receive only the tags matching their annotation coverage. This prevents the model from learning contradictory gradients (penalizing it for missing labels that were never annotated).
• Training Strategy: The same audio is trained under both policies, distinguished only by the prefix tags. This teaches the encoder to represent all available content (words, disfluencies, sounds) while the tags dictate the emission policy.

B. Supervised Cross-Attention for Word-Level Timing

Once the output token sequence is stabilized by mode tags, the authors apply supervised cross-attention to improve timing.

• Objective: Instead of relying on emergent, unsupervised alignment, they directly supervise selected cross-attention heads with word-boundary targets derived from forced alignment.
• Loss Function: They minimize the mean cosine distance between the averaged attention distribution of selected heads and a binary ground-truth target vector representing the word's time span.
• Inference: To handle the tokenizer's merging of whitespace, they decouple pause detection from attention using an energy-based pause model and apply temperature scaling (sharpening) to the attention distributions before Viterbi alignment.

C. Verbatimize: Transcript-Conditioned Disfluency Recovery

The authors introduce a new task called Verbatimize, which enables the scalable creation of verbatim transcripts from intended ones.

• Mechanism: Given an audio file and an intended transcript (as a prompt), the model reconstructs a canonical verbatim version. It copies the prompted content exactly while inserting acoustically grounded disfluencies (fillers, repetitions, cut-offs, sounds) at their temporal locations.
• Training: This involves a "casing perturbation" technique where content words in the prompt are randomly uppercased to force the model to attend to the prompt for exact spelling while relying on audio for disfluency localization.

3. Key Results

Latent Capability and Cross-Lingual Transfer

• Zero-Shot Activation: Training only the mode-tag embeddings (freezing the 764M parameter encoder and decoder) raised the German disfluency event F1 from 10% to 79% zero-shot, despite no German verbatim training data.
• Full Fine-Tuning: Full English-only fine-tuning (FT0) achieved 93.8% disfluency event F1 on German and 90.7% on English.
• Policy Control: Removing mode tags from the fine-tuned model dropped German F1 to 60.1%, demonstrating that explicit policy signals are critical for cross-lingual transfer and stability.
• Intended Quality: Mode tags successfully suppressed the leakage of disfluencies into intended output (Insertion Rate dropped from ~15% to ~3-4%) without compromising recognition accuracy (Substitution and Deletion rates remained stable).

Word-Level Timing

• Performance: The supervised cross-attention method achieved a Mean Absolute Error (MAE) of 36 ms on read speech (TIMIT) and 102 ms on disfluent speech (FluencyBank).
• Comparison: This outperforms forced-alignment baselines (e.g., MFA degrades to 142 ms on disfluent speech; WhisperX reaches 200 ms). The method also generalizes to German (55 ms MAE) with zero German timing supervision.

Verbatimize

• Rare-Word Recovery: On a rare-word evaluation set, Verbatimize improved Rare-Word Recall (RWR) from 6.8% (acoustic-only transcription) to 96.1%, with a Content Loss Rate (CLR) of only 1.3%. This confirms the model can faithfully copy prompted content while adding disfluencies.

4. Significance and Claims

The paper claims that transcription style in modern ASR is an uncontrolled latent variable causing measurable instability and evaluation errors. The significance of the work lies in:

1. Demonstrating Latent Capability: Showing that large-scale models already possess the ability to transcribe verbatim or intended text, requiring only an explicit interface (mode tags) to activate it, rather than learning these skills from scratch.
2. Resolving Evaluation Ambiguity: Providing a framework to disentangle content errors from style mismatches, allowing for fairer evaluation of conversational ASR.
3. Scalable Data Enrichment: Introducing "Verbatimize" as a method to generate high-quality canonical verbatim transcripts from abundant intended transcripts, addressing the scarcity of annotated disfluency corpora, particularly for non-English languages.
4. Unified Approach: Offering a single model that can switch between verbatim and intended modes, provide precise word-level timing on disfluent speech, and perform transcript-conditioned generation, outperforming specialized baselines in all categories.

The authors conclude that the jump from 10% to 94% zero-shot performance supports the view that mode-specific transcription largely requires activation rather than acquisition.