Technical Summary: Reference-Driven Multi-Speaker Audio Scene Generation from In-the-Wild Priors

Problem Statement

Existing multi-speaker dialogue generation systems rely heavily on structured supervision, such as per-turn speaker tags, multi-stream transcriptions, or learnable speaker-turn embeddings. These approaches typically operate within speech-only pipelines that produce clean vocal sequences, stripping away the ambient texture, room acoustics, and overlapping speech characteristic of real-world conversations. Furthermore, zero-shot voice cloning models generally address only the single-speaker case, requiring multi-speaker scenes to be assembled segment-by-segment outside the model. The core challenge is to generate rich, multi-speaker audio scenes with natural overlapping speech and environmental sounds using only a free-form natural language prompt and a set of reference voices, without requiring per-turn structural annotations or identity encoders.

Methodology: SCENA Framework

The authors propose SCENA, a flow-matching framework that conditions a pretrained text-to-audio foundation model (specifically an audio-only variant of LTX-2) on multiple reference voices and a holistic natural language prompt.

1. Reference Conditioning via Latent Concatenation

Instead of using complex adapters or identity encoders, SCENA employs a minimal interface:

• Latent Concatenation: Reference audio clips are encoded into latent sequences using the same Variational Autoencoder (VAE) as the target audio. These reference latents are concatenated directly with the noisy target tokens.
• Identity-Aware Positional Encodings: To distinguish between different references and the target, each reference slot $k$ is assigned a learned embedding $e_k$. This embedding is added to the tokens of the corresponding reference latent. The target tokens receive no such additive embedding.
• Natural Language Interface: A single text prompt describes the entire scene, including ambient sounds, speaker turns, and content. The prompt refers to speakers via textual mentions (e.g., "the speaker from reference 1"). The model relies entirely on this text to bind specific references to specific parts of the generated audio.

2. The Reference Shortcut Challenge

The authors identify a critical failure mode in training reference-conditioned flow-matching models, termed the Reference Shortcut.

• Mechanism: Under standard noise schedules (e.g., logit-normal), the training mass is concentrated on intermediate timesteps where the noisy target $z_t$ still retains significant acoustic information from the clean target $z_0$. In this regime, the model can bypass the text prompt entirely. Instead of learning to route queries from the text description to the correct reference via cross-attention and self-attention, the model simply matches the noisy target tokens to the most acoustically similar reference latent via self-attention.
• Consequence: While this shortcut minimizes training loss, it leads to catastrophic failure at inference. Inference begins at pure noise ($t=1$), where acoustic similarity is unavailable. Since the model learned to ignore the text prompt during training, it fails to bind references to the correct speakers, resulting in confused or inconsistent outputs.

3. High-Noise-Biased Timestep Distribution

To eliminate the Reference Shortcut, the authors modify the training noise schedule:

• Strategy: They replace the standard logit-normal distribution with a mixture distribution: $p(t) = (1 - \lambda) \text{Beta}(t; \alpha, 1) + \lambda \text{Uniform}(t; \epsilon, 1)$.
• Effect: The Beta component concentrates training mass in the high-noise tail (where $t$ is close to 1). In this regime, the noisy target contains little to no acoustic information about the source, making the shortcut impossible. The model is forced to rely on the text prompt and the reference embeddings to determine speaker assignment.
• Auxiliary Augmentations: To further tighten binding, the authors employ adversarial reference injection (filling unused slots with distractor references not mentioned in the prompt) and slot-shuffle augmentation (randomizing reference order and updating prompt tokens accordingly) to prevent the model from relying on positional biases.

4. Dataset Construction

The authors constructed a multi-reference dataset where each example consists of a target multi-speaker conversational clip, $K$ single-speaker reference clips, and a caption. The caption generation pipeline uses an intermediate caption (unconditioned on references) to ground the scene description, which is then combined with diarization timestamps and reference clips to generate the final reference-conditioned prompt.

Results

The method was evaluated on the CoVoMix2-Dialogue benchmark, specifically the COVOMIX2-DIALOGUE-20S subset and a COVOMIX2-DIALOGUE-WILDREF subset using in-the-wild references.

• Speaker Binding: SCENA outperforms existing multi-speaker dialog baselines (including MOSS-TTSD, VibeVoice, ZipVoice-Dialog, and Dia) on speaker-binding metrics. It achieves the best cpWER (speaker-aware Word Error Rate), cpSIM (strict per-speaker cosine similarity), and ACC (accuracy of speaker assignment).
• Robustness: In the "Wild Reference" setting, where references contain background noise and non-studio conditions, SCENA maintains superior performance. While baseline cpSIM scores dropped significantly (falling below 0.40), SCENA retained a score above 0.42.
• Audio Quality: SCENA generates overlapping speech, spontaneous paralinguistic events (laughter, sighs), and scene-level ambient sound jointly with the conversation. While UTMOS scores were slightly lower than some baselines (attributed to the LTX-2 backbone's training on in-the-wild video soundtracks rather than studio speech), SQUIM scores were competitive or superior.

Significance and Claims

The paper claims that SCENA demonstrates the advantage of using a general-purpose audio foundation model conditioned on a free-form scene description, rather than passing structured dialog scripts through a speech-only pipeline.

Key claims include:

1. Naturalness: By leveraging a model pretrained on large-scale in-the-wild data, SCENA inherits the capacity to generate natural, non-studio audio textures, including background noise, room acoustics, and overlapping dialogue, which are typically absent in structured dialogue-TTS systems.
2. Minimal Interface: The framework achieves complex multi-speaker control without per-turn tags, transcripts, or identity encoders, relying solely on natural language and reference latents.
3. Training Dynamics: The paper highlights the "Reference Shortcut" as a previously unrecognized failure mode in reference-conditioned flow matching and proposes that modifying the timestep distribution alone is sufficient to force the model to learn the necessary text-to-reference binding, without adding explicit binding mechanisms.

The authors conclude that their approach enables complex human interactions, including overlapping speech and spontaneous paralinguistic events, directly from a single natural language prompt.