In-Sync: Adaptation of Speech Aware Large Language Models for ASR with Word Level Timestamp Predictions

Technical Summary: IN-SYNC: Adaptation of Speech-Aware Large Language Models for ASR with Word-Level Timestamp Predictions

1. Problem Statement

Modern Automatic Speech Recognition (ASR) is shifting from simple transcription toward holistic speech processing. While Large Language Models (LLMs) integrated with acoustic encoders (SpeechLLMs) have achieved high transcription accuracy, most systems still rely on external, two-pass alignment tools (like Montreal Forced Aligner or WhisperX) to provide word-level timestamps. These external tools often require separate acoustic models, pronunciation dictionaries, or additional computational passes.

The core challenge addressed in this paper is how to enable a Speech-aware LLM to perform Speech Recognition with Timestamps (SRWT) in a single, end-to-end pass without sacrificing transcription accuracy (Word Error Rate) or suffering from error propagation during autoregressive generation.

2. Methodology

The authors build upon the Granite-speech-8B framework, which utilizes a 10-layer Conformer audio encoder, a task-aware MLP adapter, and a Granite-3.3-8B-Instruct LLM. The proposed In-Sync framework employs a multi-task learning approach where the model is trained on both ASR and SRWT tasks.

To stabilize this joint training, the authors introduce three novel lightweight strategies:

• Speech Length Augmentation: To combat the "heavy-tailed" distribution of timestamps (where short durations are over-represented and long durations are rare), the authors concatenate consecutive utterances during training. This forces the model to learn and predict larger, long-range timestamp tokens.
• Timestamp Embedding Regularization: To ensure the model understands that time is monotonic, the authors introduce an auxiliary loss. This loss uses a Gaussian kernel to enforce a structured similarity matrix among timestamp embeddings, encouraging neighboring time tokens to have higher cosine similarity than distant ones.
• Reduced Teacher Forcing: In standard autoregressive training, the model always sees ground-truth previous tokens. This can lead to "cascading failures" where one wrong timestamp ruins the entire sequence. The authors randomly corrupt timestamp tokens in the input during training, forcing the model to rely on both the global audio context and local temporal cues rather than just the previous (potentially erroneous) timestamp.

3. Key Contributions

• Unified Framework: An end-to-end architecture that performs transcription and word-level timestamping simultaneously, eliminating the need for external aligners.
• Task-Aware Adaptation: The use of a task-specific indicator token in the speech adapter to allow the model to generate distinct representations for ASR vs. SRWT.
• Robustness Strategies: Three specific training techniques (Length Augmentation, Embedding Regularization, and Reduced Teacher Forcing) designed to solve the unique mathematical and distributional challenges of temporal prediction in LLMs.

4. Results

The experiments were conducted across multiple datasets (LibriSpeech, CommonVoice, AMI, VoxPopuli, etc.). Key findings include:

• Performance Trade-offs: Naive multi-task training (Mixed Training) improves timestamp accuracy but degrades ASR performance (higher WER).
• Effectiveness of Strategies:
  • Timestamp Regularization was highly effective at balancing the two objectives, improving WER while simultaneously reducing Accumulated Averaging Shift (AAS).
  • Reduced Teacher Forcing yielded the strongest overall timestamp accuracy and improved the model's robustness to error propagation.
• Comparison to Baselines: The In-Sync model outperformed Qwen-Audio in terms of task following (Qwen-Audio often failed to generate timestamps on certain datasets) and achieved a better average AAS score than CrisperWhisper.

5. Significance

This work demonstrates that timestamp prediction can be treated as a "first-class objective" within the Speech-LLM paradigm. By integrating temporal grounding directly into the language modeling process, the researchers move closer to a truly unified speech-to-text system. This has significant implications for real-world applications requiring high-precision synchronization, such as automated video captioning, media search, and multimodal AI interaction, where both what was said and when it was said are equally critical.