DuplexChat: Constructing Speaker-Separated Full-Duplex Dialogue Speech at Scale for Spoken Dialogue Language Modeling
The paper introduces DuplexChat, a large-scale open-source corpus of speaker-separated full-duplex dialogue speech in English and Japanese, constructed via the DuplexChat-Pipe pipeline from public podcasts to address the lack of suitable training data for spoken dialogue language models.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you want to teach a robot how to have a natural, human-like conversation. The problem is that most of the "textbooks" we have for teaching robots are like listening to a single person talking on a phone call where you can't tell who is speaking when two people are talking at once. It's a messy, mixed-up audio track.
To teach a robot the art of real conversation—where people interrupt each other, say "uh-huh" while the other person is talking, and jump back and forth quickly—you need a special kind of training data: two separate audio streams, one for each person, perfectly synchronized.
This paper introduces DuplexChat, a massive new library of such conversations, and DuplexChat-Pipe, the magical machine that built it.
The Problem: The "Smoothie" vs. The "Fruit Bowl"
Think of existing public speech data (like podcasts) as a giant fruit smoothie. You have apples (Speaker A), oranges (Speaker B), and maybe some ice cream (music or ads) all blended together into one single cup. You can taste the fruit, but you can't separate the apple from the orange to study them individually.
For a robot to learn natural conversation, it needs a fruit bowl where every piece of fruit is in its own separate bowl. Until now, getting this "fruit bowl" at a massive scale was nearly impossible because it usually required hiring people to record phone calls, which is slow and expensive.
The Solution: The "DuplexChat-Pipe" Factory
The authors built an open-source factory called DuplexChat-Pipe that takes the "smoothie" (public podcast feeds from the internet) and magically sorts it back into a "fruit bowl." Here is how the factory works, step-by-step:
- Finding the Ingredients (Feed Collection): The factory scans the internet for podcast RSS feeds (like a shopping list of shows) and filters out the ones that aren't in English or Japanese.
- Washing the Fruit (Audio Retrieval & Cleaning): It downloads the audio episodes. If a show is just music, or if an episode is longer than three hours (usually a long music stream), it gets thrown out. The remaining audio is cleaned up and standardized.
- Cutting the Right Pieces (Dialogue Segmentation): The factory uses a smart "speaker detector" (diarization) to find the parts of the podcast where exactly two people are talking. It cuts out the music, the ads, and the parts where only one person is monologuing. It keeps only the "two-person chat" clips.
- The Magic Separation (Speech Separation & Restoration): This is the most critical step. The factory uses a special AI model (DialogueSidon) to take the mixed audio (the smoothie) and split it back into two separate tracks: one for the person on the left and one for the person on the right. It also cleans up the noise, making the voices crisp and clear.
The Result: A Massive Library of Conversations
By running this pipeline, the authors created DuplexChat, which is currently the largest resource of its kind in the world.
- English: Over 282,000 hours of two-person conversations.
- Japanese: Over 132,000 hours of two-person conversations.
To put that in perspective, previous large datasets were like a small library; DuplexChat is like the entire Library of Congress.
Does It Work? (The Taste Test)
The authors checked if their "fruit bowl" was actually good quality by comparing it to the gold standard, a dataset called Fisher (which consists of real phone calls).
- Sound Quality: The voices in DuplexChat are actually cleaner and less noisy than the phone call recordings.
- Separation: The AI did a great job of keeping the two voices distinct, though it worked slightly better for English than Japanese (likely because the separation tool was trained more on English data).
- Realism: They analyzed how people talk in these recordings. They found that the conversations have the natural "dance" of human speech: people interrupt each other, use backchannels (like "yeah" or "right"), and switch turns quickly. The Japanese data, for instance, showed even more frequent interruptions and overlapping speech, which matches real human behavior in that culture.
The Bottom Line
The paper claims that DuplexChat-Pipe is the first open, reusable tool that can turn the chaotic, mixed audio of the internet into a clean, separated, two-person conversation dataset at a massive scale. The resulting DuplexChat dataset provides the "fruit bowl" needed to train the next generation of AI that can hold a natural, back-and-forth conversation with humans, capturing the messy, overlapping, and lively dynamics of real speech.
Note: The authors are careful to state that because they scraped this data from the public internet, they only released the "links" to the audio and the code to rebuild it, rather than the audio files themselves, to respect copyright laws.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.