💻 computer science

DuplexGen: Decoupling Content, Timing, and Acoustics for Synthetic Dialogue Speech

DuplexGen is a novel dialogue synthesis framework that decouples content, timing, and acoustics by using an LLM for script generation followed by two full-duplex conversational models interacting in real time, thereby enabling naturally emerging conversational dynamics like interruptions and overlaps while preserving scripted content and high-fidelity speech quality.

Original authors: Pengcheng Wang, Sheng Li, Jiyi Li, Takahiro Shinozaki

Published 2026-08-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Pengcheng Wang, Sheng Li, Jiyi Li, Takahiro Shinozaki

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Human conversation is a delicate, high-speed dance of timing, where speakers constantly listen while they speak, interrupting one another, overlapping in brief bursts, and offering quick acknowledgments like "mm-hmm" without breaking the flow. For decades, computers have struggled to mimic this natural rhythm. While artificial intelligence has become excellent at generating the words people say, it has historically failed to capture the messy, spontaneous timing of real talk. Most existing systems generate a script first and then try to force interruptions or pauses into the audio using rigid, pre-written rules. The result often sounds stiff and unnatural, lacking the fluid give-and-take that defines a genuine human exchange. This limitation matters deeply for researchers building voice recognition systems; if the training data sounds too perfect or too robotic, the software will fail when it encounters the chaotic reality of a real doctor's office or a busy café.

A team of researchers has introduced a new approach called DuplexGen that changes how synthetic conversations are built. Instead of treating dialogue as a single, monolithic process, they separated the task into three distinct stages: deciding what is said, deciding when it is said, and deciding how it sounds. First, a large language model writes the script, determining the exact words and the order of speakers. This ensures the content remains fixed and controllable. Next, two artificial intelligence models act out this script simultaneously. Unlike traditional systems that follow a strict schedule, these models listen to each other in real time, just as humans do. They decide independently whether to speak the next word from the script or to stay silent, allowing the timing of the conversation to emerge naturally from their interaction. Finally, a high-fidelity voice synthesizer re-records the entire interaction, preserving the exact timing and overlaps created by the two models while ensuring the audio sounds clear and realistic.

The researchers tested this method by creating a corpus of simulated conversations between patients and clinicians, a setting where precise timing and natural interruptions are critical. They compared their emergent, interaction-driven approach against older methods that simply stitched together pre-recorded speech segments with fixed or random gaps. The results showed a clear difference. The new method produced conversations with overlapping speech and long pauses that closely matched the statistical patterns found in real human recordings. In contrast, the older stitching methods failed to reproduce these natural dynamics, collapsing into a single, unnatural pattern of speech with almost no overlap. The study found that the new framework reduced the statistical distance between synthetic and real conversation timing by nearly half, successfully capturing the complex rhythm of human interaction without altering the scripted content.

Beyond just sounding more natural, this approach offers a powerful tool for testing voice recognition technology. Because the researchers controlled the timing and the content separately, they could create specific levels of difficulty for the software to solve. They generated three tiers of conversation: polite interactions with few interruptions, natural interactions with typical overlap, and adversarial interactions with dense, frequent interruptions. When they tested popular voice recognition systems on these recordings, the error rates increased steadily as the conversation became more overlapping and difficult. Crucially, the study revealed that the difficulty came from the timing of the overlap itself, not just the quality of the audio. This distinction allows engineers to stress-test their systems in a controlled way, ensuring they can handle the messy reality of two people talking over each other.

The researchers acknowledge that their system is not a perfect replica of human conversation. Because the words are fixed before the interaction begins, the speakers cannot change their minds or correct themselves mid-sentence, which limits the depth of the interaction. Additionally, the current system handles overlaps primarily at the boundaries of sentences rather than deep within them. However, the work demonstrates that by decoupling content from timing, it is possible to generate synthetic speech that feels alive and responsive. This breakthrough provides a new, reliable way to create the vast amounts of realistic training data needed to teach computers how to understand the complex, overlapping nature of human speech.

Technical Summary: DuplexGen – Decoupling Content, Timing, and Acoustics for Synthetic Dialogue Speech

Problem Statement

Synthetic conversational speech is a critical resource for developing and evaluating conversational speech recognition (ASR) and speaker diarization systems. However, existing dialogue synthesis pipelines suffer from a fundamental limitation: they typically generate dialogue content first and subsequently insert conversational dynamics (such as interruptions, overlaps, and backchannels) using handcrafted markers or fixed timing rules. This approach results in conversational timing that is prescribed by the generation pipeline rather than emergent from interaction.

Conversely, recent full-duplex conversational models can generate natural interaction dynamics through continuous listening while speaking, but they generate content freely and cannot reliably adhere to a predefined dialogue script. Current approaches struggle to combine controllable content with emergent interaction, leaving this combination underexplored.

Methodology: The DuplexGen Framework

The authors propose DuplexGen, a framework that explicitly decouples dialogue generation into three independent stages: content, timing, and acoustics. This separation ensures that each stage is responsible for exactly one aspect of the conversation.

1. Content Generation (Scripting)

A Large Language Model (LLM) generates the dialogue script, determining the lexical content and the order of speakers. This stage fixes the "what" of the conversation.

2. Timing Generation (Script-Constrained Duplex Decoding)

Two full-duplex conversational models perform the script in real-time by listening to each other. To ensure the content remains fixed while allowing timing to emerge, the authors introduce a script-constrained duplex decoding strategy:

Lexical Constraint: The model predicts text tokens before generating audio. At each frame, the candidate vocabulary is restricted to either the next script token (if the speaker is eligible to take the floor), a set of listener backchannels (e.g., "mm-hmm"), or a padding token (silence).
Decision Process: The model does not decide what to say; it decides when to emit the next script token or remain silent.
Control Parameters: Three parameters regulate the interaction without dictating exact timestamps:
- Handoff Window ( $W_h$ ): Defines the interval around the projected end of a turn where a listener may take the floor.
- Max Padding ( $P_{max}$ ): Limits consecutive silence frames to prevent unrealistic gaps.
- Backchannel Refractory ( $R_{bc}$ ): Enforces a minimum interval between listener backchannels.
Emergence: Floor transitions, overlap durations, and backchannel attachment points are determined online by the interaction between the two models, allowing conversational dynamics to emerge naturally.

3. Acoustic Realization (Fidelity Re-rendering)

The duplex models output a symbolic interaction score (recording active speakers, turn boundaries, overlaps, and backchannels) rather than a final waveform. A high-fidelity Text-to-Speech (TTS) system (CosyVoice) re-renders this interaction:

Timing Transfer: Absolute timestamps are not copied directly. Instead, relative positions are preserved. For overlaps, the onset of the next turn is placed at a specific fraction of the current turn's duration.
Acoustic Realism: Interrupted speech is physically truncated with a short crossfade to simulate real interruptions. Backchannels are synthesized independently and mixed into the main conversation at their mapped positions.
Automatic Annotation: Because the symbolic score fully specifies the conversation, annotations (word timestamps, speaker activity, overlap regions) are generated automatically during synthesis.

Key Contributions

Decoupled Framework: A dialogue synthesis framework that explicitly separates content generation, conversational dynamics, and acoustic rendering.
Script-Constrained Duplex Decoding: A strategy that enforces exact lexical constraints during duplex generation while allowing overlap, interruptions, and backchannels to emerge naturally through real-time interaction.
Interaction-Aware Corpus: The construction of an interaction-aware patient–clinician conversational speech corpus with construction-time annotations, demonstrating that the framework produces dynamics closer to real dialogue distributions than stitching baselines.

Experimental Results

The authors evaluated the framework against stitching-based baselines (fixed gaps and random gaps) using real clinical dialogue (PriMock57) as a reference.

Distributional Realism: The proposed method significantly outperformed stitching baselines in matching real dialogue statistics.
- Floor-Transfer Offset (FTO): The Wasserstein distance ( $W_{fto}$ ) improved from 0.695 (random stitching) to 0.366 (DuplexGen), a 47% relative improvement.
- Overlap and Gaps: Unlike stitching baselines which collapsed to a single near-zero-gap mode, DuplexGen naturally reproduced both overlapped transitions (38.5% vs. 42.8% real) and long-tail silence distributions (30.8% vs. 20.5% real).
Fidelity: The decoupling was verified to be effective. Content fidelity was perfect (0% forced WER), and overlap events were transferred exactly. Acoustic re-rendering introduced minimal error (≤1.7% WER) while maintaining speaker consistency (CAM++ similarity of 0.82).
Controllability: The system adapted to different dialogue styles (medical vs. casual) under identical timing parameters, producing less overlap in medical contexts (7.2%) compared to casual ones (10.5%), closely matching real clinical behavior (6.3%).
Downstream Stress Test: A benchmark of 180 segments (175 minutes) was created to test ASR robustness. Results showed that overlap-region errors were 3–7× higher than aggregate errors and increased monotonically with interaction difficulty.
- Ablation Study: A 2×2 ablation revealed that acoustic rendering (specifically the intensity of overlap) drives downstream ASR difficulty more than the timing mechanism itself. However, the emergent timing successfully regulated the difficulty levels.

Significance and Claims

The paper claims that DuplexGen offers a "zero-training pipeline" that allows overlapping conversational speech to be scripted for content yet emergent in timing and high-fidelity in sound.

Realism: The emergent turn-taking is distributionally closer to real clinical dialogue than stitching methods, successfully reproducing complex dynamics like overlapped transitions and long gaps that are absent in baselines.
Utility: The resulting framework provides a controllable benchmark for stress-testing ASR and speaker diarization systems, complete with construction-time ground truth.
Modesty: The authors acknowledge limitations, noting that interaction is only partially emergent (global statistics are still regulated by parameters), that deep mid-utterance interruptions are not yet modeled, and that the strict decoupling limits adaptability (speakers cannot revise content after interruption). They also note that the current benchmark is a simplified approximation of real clinical conversations, where acoustic rendering influences difficulty more than emergent timing.

The work is positioned as a step toward more realistic synthetic dialogue data, bridging the gap between controllable content generation and natural interaction modeling.