Technical Summary: LiveBand – Live Accompaniment Generation in the Audio Domain

1. Problem Definition

The paper addresses the challenge of real-time music accompaniment generation from a live audio input stream. The core difficulty lies in strict causality: the model must generate high-fidelity accompaniment frames at low latency without access to future input (the "future mix").

Existing approaches face two primary limitations in this setting:

1. Exposure Bias and Error Accumulation: Traditional autoregressive models trained with "teacher forcing" (feeding ground-truth past frames during training) suffer from a distribution shift during inference. When the model generates its own past predictions as input, small errors accumulate, leading to long-horizon drift and musical asynchrony.
2. Frame-Level vs. Sequence-Level Objectives: Standard frame-level prediction objectives rigidly penalize minor anticipatory errors. However, in a real-time setting where future context is unknown, even expert musicians make small timing adjustments. Rigid frame-level objectives fail to capture the holistic coherence required for a natural jam session.
3. Lookahead Dependency: Recent work suggests that strong adherence between generated accompaniment and the input mix may require "lookahead" (accessing future mix frames), which violates strict real-time constraints.

2. Methodology: LiveBand

LiveBand is a real-time system that generates accompaniments in the continuous latent space of a pre-trained causal audio autoencoder. The architecture and training paradigm are designed to eliminate the mismatch between training and inference.

2.1 Core Components

• Causal Audio Autoencoder (AE): A pre-trained, strictly causal variant of CoDiCodec. It encodes stereo waveforms (44.1 kHz) into continuous latent sequences (~10 Hz, dimension 128) with a 4096× downsampling ratio. Neither the encoder nor decoder accesses future audio context.
• Generator (G): A causal transformer operating on latent frames.
  • Inputs: At each timestep $t$, it receives a causally available mix history ($m_{\le t}$) and an independent Gaussian noise vector ($z_t$).
  • Architecture: Pre-norm transformer blocks with adaptive layer normalization, causal multi-head self-attention (using query-key normalization and Rotary Position Embeddings), and SwiGLU feed-forward networks.
  • Mechanism: It predicts a future accompaniment frame $a_{t+1+\delta}$ (where $\delta$ is anticipation in latent frames) without accessing future mix information or ground-truth target latents.
• Discriminator (D): A non-causal 1D convolutional network (used only during training) that scores the realism of a complete accompaniment sequence conditioned on the corresponding mix. It evaluates the joint sequence of mix and accompaniment latents.

2.2 Training Paradigm: Eliminating Teacher Forcing

The central innovation is the removal of teacher forcing to ensure training-inference equivalence:

• No Ground-Truth Feedback: During training, the generator never receives ground-truth past accompaniment frames as input. Instead, it relies on the internal self-attention over hidden states and the independent noise vector at each step.
• Parallel Causal Pass: Because the generator does not depend on its own previous outputs, the entire sequence can be generated in a single parallel forward pass under a causal mask. This is computationally identical to the step-by-step autoregressive inference process (using a KV cache), thereby eliminating exposure bias by design.
• Sequence-Level Adversarial Supervision: Instead of minimizing frame-wise prediction error, the model is trained using a Relativistic GAN (R3GAN) objective. The discriminator evaluates the entire generated sequence against the real sequence, providing distribution-level supervision that encourages global musical coherence and tolerates minor local timing deviations.

2.3 Stabilization Techniques

• Attention Sinks: To prevent drift in long-form streaming caused by KV cache eviction, the model employs a scalar attention sink mechanism, ensuring initial tokens remain accessible.
• Adaptive Gradient Penalty (AdaGP): An online mechanism dynamically adjusts the gradient penalty weight ($\lambda$) in the discriminator's loss to maintain a fixed target advantage ($a^*$) of the discriminator over the generator. This stabilizes adversarial training without manual hyperparameter tuning.

3. Key Contributions

1. Strictly Causal Transformer with Adversarial Supervision: The introduction of a model trained with sequence-level adversarial supervision that operates without lookahead, achieving strong mix adherence.
2. Training-Inference Equivalence: The first streaming accompaniment model to fully match training and inference conditions by design, eliminating exposure bias and error accumulation without requiring expensive sequential rollouts during training.
3. Adaptive Gradient Penalty: A novel method (AdaGP) that automatically maintains discriminator stability, removing the need for manual tuning of gradient penalty weights.

4. Experimental Results

The model was evaluated on the Slakh2100 dataset (and an internal corpus for a CLAP-conditioned variant) against StreamMusicGen (SMG), a state-of-the-art autoregressive baseline.

• Objective Metrics: LiveBand outperforms SMG across all metrics (FAD, Beat Alignment, COCOLA) in both synchronous ($\tau=0$) and anticipatory ($\tau=0.1s, 1s$) settings.
  • Drift: Unlike SMG, which shows performance degradation over time, LiveBand exhibits near-zero or favorable drift, indicating it does not accumulate local errors.
  • Anticipation: LiveBand maintains high performance even when generating 1 second ahead of the input mix, a regime where SMG struggles.
• Subjective Listening Tests: In a study with 19 participants, LiveBand was significantly preferred over SMG in audio quality, temporal consistency, and mix adherence.
• Real-Time Performance: On a consumer RTX 3090 GPU, LiveBand achieves a Real-Time Factor (RTF) of 1.1× (eager mode) and 2.1× (compiled mode) for a 0.1s anticipation budget, confirming deployability on consumer hardware.

5. Significance and Claims

The paper argues that the limitation of requiring future lookahead for high-quality live accompaniment is not fundamental to the task but rather a result of supervised learning mismatches (teacher forcing and frame-level objectives).

• Resolution of Drift: By replacing next-step predictive supervision with sequence-level adversarial supervision and removing teacher forcing, the model can generate ahead of the live input without error accumulation.
• Holistic Coherence: The adversarial objective encourages the model to preserve long-range musical structure, allowing it to tolerate the small, natural timing corrections required in real-time performance.
• Feasibility: The work demonstrates that robust, interactive, real-time generative models can be built using sequence-level adversarial training, enabling musicians to jam with AI companions that adapt to tempo, key, and dynamics without latency-induced asynchrony.

The authors acknowledge that while the system achieves strong musical coherence, the audio quality (fidelity) still leaves room for improvement compared to ground truth, suggesting future work should focus on higher-fidelity causal autoencoders.