⚡ electrical engineering

LibriConvo: Simulating Conversations from Read Literature for ASR and Diarization

This paper introduces LibriConvo, a 240-hour synthetic conversational speech corpus constructed by enhancing the Speaker-Aware Simulated Conversation framework with realistic timing and acoustic properties, which serves as a practical benchmark demonstrating superior performance in speaker diarization and ASR compared to existing pipelines and models.

Original authors: Máté Gedeon, Péter Mihajlik

Published 2026-06-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Máté Gedeon, Péter Mihajlik

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand a crowded room where two people are having a lively, overlapping conversation. The robot needs to do two things at once: figure out who is speaking at any given moment (Speaker Diarization) and what they are saying (Automatic Speech Recognition).

The problem is that real conversations are messy, expensive to record, and hard to label perfectly. So, scientists often build "fake" conversations by gluing together pre-recorded sentences. But usually, these fake conversations feel robotic—people pause too long, the timing is off, and the sentences don't make sense together.

This paper introduces LibriConvo, a new, massive library of "fake" but highly realistic conversations designed to train these robots better. Here is how they built it, using simple analogies:

1. The "Script" Problem: Making Sense of the Chaos

In the past, researchers would grab random sentences from different books and mash them together. It's like trying to have a conversation where one person quotes Harry Potter and the other quotes The Great Gatsby. The robot gets confused because the context is broken.

The Fix: The authors decided to stick to one book per conversation. They took sentences from LibriTTS (a database of people reading books) and only used sentences from the same story for a single dialogue.

Analogy: Instead of a chaotic mix of random quotes, they created a "book club" simulation where both speakers are discussing the same chapter. This helps the robot understand the flow of the story, making it easier to recognize words.

2. The "Timing" Problem: Fixing the Awkward Pauses

When researchers tried to simulate how people talk, they found the pauses were too long and awkward. It felt like a conversation between two people who were constantly waiting for the other to finish, even when they weren't.

The Fix: They analyzed real phone calls (CallHome) to see how people actually pause. They found the original data was messy, so they used a smart tool to find the exact start and end of speech. Then, they applied a "time-compression" filter.

Analogy: Imagine a video of a conversation where the editor accidentally left in 5 seconds of silence between every sentence. The authors ran a "fast-forward" button that only speeds up the long, awkward silences while keeping the natural, quick pauses intact. This makes the conversation flow like a real human chat.

3. The "Room" Problem: Where are they standing?

When you record a conversation, the sound changes depending on where you are standing and what the room looks like. Previous simulations often put speakers in weird spots, like floating in the middle of a ceiling or right against a wall, which doesn't happen in real life.

The Fix: They used a database of real room sounds (Room Impulse Responses) but added a "common sense" filter.

Analogy: Imagine a director casting actors for a scene. They wouldn't put the actors standing on the ceiling or inside a wall. The authors created a rule that only allows speakers to stand in "human-sized" spots (about 1.5 meters high, 1 meter apart) and at different angles, ensuring the sound feels like it's coming from a real living room, not a sci-fi movie set.

4. The Result: A Giant Training Gym

The final product, LibriConvo, is a massive dataset containing 240 hours of these simulated conversations.

It has 1,496 dialogues involving 830 different speakers.
Crucially, they split the data so that the speakers in the "training" group are completely different from the speakers in the "testing" group. This ensures the robot is learning to recognize anyone, not just memorizing specific voices.

5. Did it Work? (The Scoreboard)

The authors tested two types of robots on this new dataset:

The "Who is Speaking?" Test (Diarization):
They compared a standard tool (pyannote) against a newer, smarter tool called Sortformer.
- Result: Sortformer was much better. It made fewer mistakes (11.1% error rate vs. 24.4%).
- Why? It's better at untangling the mess when two people talk over each other, kind of like a skilled conductor keeping track of two musicians playing at the same time.
The "What Did They Say?" Test (Speech Recognition):
They tested big, pre-trained models (like Whisper) against a custom-trained model (FastConformer).
- Result: The custom-trained model won. It achieved a very low error rate (6.97%).
- Why? They taught the model a special trick called "Serialized Output Training." Instead of chopping up overlapping speech into tiny, confusing pieces, the model learned to keep each speaker's sentence whole, even if they talked over each other. It's like listening to a duet and writing down the lyrics for the singer on the left, then the singer on the right, without getting the words mixed up.

Summary

The paper doesn't claim this solves every problem in the world yet, but it provides a better training ground. By fixing the timing, the story context, and the room acoustics, they created a dataset that helps robots learn to handle messy, real-world conversations much more effectively than before. It's a new, high-quality "gym" for speech AI to get stronger.

Technical Summary: LibriConvo – Simulating Conversations from Read Literature for ASR and Diarization

Problem Statement

Modern speech processing systems, particularly end-to-end speaker diarization (EEND) and multi-speaker automatic speech recognition (ASR), require large volumes of annotated conversational audio. However, collecting real multi-party conversations with reliable turn-level labels is expensive and difficult. While synthetic data generation has emerged as a common alternative, naive utterance mixing often fails to capture realistic turn-taking behaviors, stable speaker identity patterns, and semantic continuity across a full dialogue. Furthermore, previous frameworks like Speaker-Aware Simulated Conversation (SASC) provided the theoretical modeling of conversational timing but lacked a released benchmark dataset and downstream evaluation results for ASR and diarization tasks.

Methodology

The paper introduces LibriConvo, a synthetic conversational speech corpus constructed by instantiating the SASC framework with specific enhancements designed to improve realism and suitability for downstream tasks. The construction pipeline involves several key technical components:

1. Conversational Timing and Statistics

To address inconsistencies in existing annotations (specifically regarding overlap and gap statistics in the Switchboard corpus), the authors utilized the English CallHome corpus. They employed an external Voice Activity Detection (VAD) model (Silero VAD) to derive more reliable temporal boundaries.

Pause Compression: Preliminary analysis revealed that reconstructed dialogues contained unnaturally long pauses. To emulate spontaneous speech, a deterministic piecewise linear transform was applied to silence durations. This selectively compresses long silences (using a compression coefficient $s=0.4$ for pauses exceeding a threshold $t=1.0$ ) while preserving short gaps, thereby maintaining the relative ordering of events while reducing artificial disjointedness.
Turn-Taking: A first-order Markov chain, trained on CallHome data, models the probability of speaker transitions.

2. Semantic Continuity

To address the lack of semantic coherence in prior simulated datasets, which can hinder ASR models relying on textual context, the authors constrained utterance selection.

Source Material: Utterances were drawn from LibriTTS, which consists of read book chapters.
Book Constraint: Each simulated conversation is constructed using utterances from a single book, ensuring that the dialogue participants share a common semantic ground.
Length Filtering: Utterances were filtered to a realistic range of 2 to 10 seconds.

3. Acoustic Realism (Room Impulse Responses)

The authors implemented a spatial-plausibility heuristic for selecting Room Impulse Responses (RIRs) from the BUT Speech@FIT Reverb Database.

Selection Strategy: Instead of random selection, configurations were ranked based on geometric attributes (height, distance, elevation, azimuth) relative to idealized human speech positions (e.g., 1.5m source height, 1m source-mic distance).
Diversity: The selection process ensures spatial diversity by requiring speakers to have azimuths differing by at least 20°, avoiding collinearity and approximating natural conversational scenes. This procedure was applied to 40% of the conversations.

4. Dataset Construction and Splits

The resulting corpus contains 240.1 hours of audio across 1,496 dialogues involving 830 unique speakers.

Speaker-Disjoint Splits: To ensure robust evaluation and prevent data leakage, the dataset was partitioned into training, validation, and test sets such that no speaker appears in more than one split.
Distribution: The splits follow an approximate 80–10–10% ratio (Train: 193.7h/580 speakers; Validation: 23.1h/127 speakers; Test: 23.3h/123 speakers).

Key Contributions

LibriConvo Dataset: The release of a 240.1-hour synthetic conversational dataset with turn-level speaker labels and transcriptions, specifically designed for both ASR and speaker diarization.
Enhanced Construction Pipeline: A dataset-oriented instantiation of the SASC framework featuring:
- Book-constrained utterance sampling to improve semantic continuity.
- A realism-driven RIR selection strategy to enhance acoustic plausibility.
- Temporal compression of pauses to better match spontaneous speech statistics.
Reproducible Baselines: The establishment of benchmark results for both diarization and ASR on speaker-disjoint splits, providing a standard for evaluating multi-speaker speech processing systems.

Experimental Results

Speaker Diarization

Two state-of-the-art frameworks were evaluated without additional fine-tuning:

Pyannote Pipeline: A modular framework using segmentation, embedding extraction, and clustering.
Sortformer: An encoder-based model using a "Sort Loss" to enforce consistent speaker label ordering.

Results: Sortformer significantly outperformed the Pyannote pipeline on the test set, achieving a Diarization Error Rate (DER) of 11.1% compared to 24.4% for Pyannote. Sortformer also demonstrated higher consistency across recordings with lower variance in error distribution.

Automatic Speech Recognition (ASR)

The authors evaluated pre-trained models (Whisper-large-v3, Canary-1B-v2) and fine-tuned FastConformer models (Large and XLarge) using Serialized Output Training (SOT).

Training Approach: Unlike token-level variants that split utterances at every overlap, the SOT approach preserves each speaker's utterance as a coherent unit, marking speaker changes with a <sc> token.
Metrics: Word Error Rate (WER), concatenated minimum-permutation WER (cpWER), and Segment Accuracy (SegAcc).

Results:

Pre-trained models performed well without fine-tuning (e.g., Whisper-large-v3 achieved 7.46% WER on the test set).
The FastConformer-CTC XLarge model, fine-tuned with SOT, achieved the best overall performance: 7.29% WER and 6.97% cpWER on the test set, outperforming the zero-shot Whisper-large-v3 baseline.
All models showed improvement when evaluated with cpWER, suggesting that while content was accurate, the ordering of speaker segments sometimes differed from references.

Significance and Claims

The paper positions LibriConvo as a practical benchmark for studying synthetic conversational speech and evaluating multi-speaker systems. The authors claim that the dataset addresses critical limitations of earlier simulated datasets, specifically the lack of semantic coherence and the presence of implausible temporal gaps.

The results demonstrate that:

End-to-end transformer architectures (like Sortformer) are highly effective at disentangling overlapping speech and maintaining speaker continuity compared to modular pipelines.
Serialized Output Training combined with fine-tuning is an effective strategy for multi-speaker ASR, preserving utterance-level coherence rather than fragmenting speech at every overlap.
The methodology is reproducible and can be extended to generate larger-scale datasets or adapted for different acoustic conditions.

The authors conclude that LibriConvo serves as a valuable resource for future research in multi-speaker speech processing, offering a realistic simulation environment that bridges the gap between theoretical frameworks and practical system evaluation.