Technical Summary: SwiftAudio

Problem Statement

Diffusion-based text-to-audio (TTA) models currently achieve high-fidelity synthesis but suffer from significant inference latency due to the requirement for iterative multi-step denoising. While recent advances in consistency models and distillation have enabled one-step generation, existing approaches face two critical limitations:

1. Quality Degradation: Strict one-step inference often results in noticeable quality drops compared to multi-step systems.
2. Data Dependency: Current one-step distillation methods still require paired text–audio datasets during training. This is a bottleneck because high-quality, manually annotated audio-caption datasets (e.g., AudioCaps with ~45K examples) are scarce and expensive compared to the millions of text prompts available for image generation.

The central question addressed by this work is whether a high-quality one-step TTA generator can be learned using only text captions and a pretrained diffusion teacher, without requiring paired audio supervision during the distillation process.

Methodology

The authors propose SwiftAudio, a framework that performs audio-free distillation from a pretrained diffusion teacher using only text prompts. The methodology consists of three core components:

1. Audio-Free Distillation via Variational Score Distillation (VSD)

SwiftAudio adapts Variational Score Distillation (VSD) to the audio domain. Instead of relying on ground-truth audio, the student model ($f_\theta$) is trained to minimize the Kullback–Leibler (KL) divergence between its generated latent distribution and the prior of a frozen teacher model ($\epsilon_\psi$).

• Mechanism: The student generates a clean latent $\hat{x}_0$ from noise $z$ and text $y$. An intermediate noisy state $x_t$ is sampled. The student is updated using the score difference between the frozen teacher and a lightweight, LoRA-adapted teacher ($\epsilon_\phi$) that tracks the student's distribution.
• LoRA Teacher: A LoRA-adapted version of the teacher is alternately trained on student-generated samples to provide accurate corrective gradients for the VSD objective, eliminating the need for paired audio data.

2. Temporal Smoothness Regularization

To address the instability and lack of coherence often found in extreme one-step generation, the authors introduce a temporal Total Variation (TV) regularization objective ($L_{temp}$).

• Rationale: Acoustic content typically evolves smoothly, with abrupt changes (e.g., percussive onsets) occurring sparsely.
• Implementation: The loss applies an $L_1$ norm to the temporal differences of the latent representation. This encourages piecewise-smooth latent trajectories, suppressing spurious frame-to-frame fluctuations in stationary regions while preserving significant transient events.

3. Joint Optimization

The student network is optimized using a total objective function:
$$L_{total} = \lambda \cdot L_{temp} + L_{VSD}$$
where $\lambda$ balances the temporal smoothness constraint against the score distillation signal. Training alternates between updating the student and adapting the LoRA teacher.

Key Contributions

1. SwiftAudio Framework: A one-step TTA generation framework that distills a pretrained diffusion teacher using only text captions, removing the dependency on paired audio-caption data during distillation.
2. Domain Adaptation of VSD: The successful adaptation of Variational Score Distillation to the audio domain, combined with a novel temporal-aware latent regularization objective that provides an audio-specific inductive bias for stable generation.
3. Data Efficiency: Empirical demonstration that caption-only distillation is highly effective in the TTA setting, achieving strong performance with only ~45K captions (from AudioCaps), a scale significantly smaller than the millions of prompts typically required for similar image-domain distillation.
4. State-of-the-Art Performance: The model achieves the best performance among strict one-step methods and substantially narrows the quality gap to multi-step diffusion systems.

Experimental Results

Experiments were conducted on the AudioCaps and Clotho datasets.

• Performance on AudioCaps: SwiftAudio outperforms existing one-step methods (AudioLCM, ConsistencyTTA) on objective metrics (FD, FAD, IS, KL). Notably, it achieves an FD of 22.73, which is nearly identical to its multi-step teacher (Auffusion-full, FD 22.49), despite reducing inference to a single forward pass (200x fewer network evaluations).
• Zero-Shot Generalization: On the out-of-domain Clotho dataset, SwiftAudio demonstrates superior transferability compared to ConsistencyTTA, which suffers significant performance degradation. This suggests SwiftAudio learns a more general text-conditioned generative prior rather than overfitting to the training domain.
• Ablation Studies:
  • Removing the diffusion-style student parameterization causes a severe performance drop.
  • Insufficient capacity in the LoRA teacher (low rank) leads to inaccurate supervision and poor results.
  • Removing temporal regularization degrades perceptual metrics (FAD increases from 2.25 to 3.47). The proposed $L_1$-based TV regularization outperforms $L_2$-based smoothness constraints by better preserving transient structures.
• Data Scaling: Performance improves with caption quantity, but the model achieves strong results with the full 45K AudioCaps set, confirming its data efficiency.

Significance and Claims

The paper claims that SwiftAudio represents a significant step forward in making text-to-audio generation both inference-efficient (single-step) and data-efficient (caption-only).

• Paradigm Shift: It demonstrates that high-quality one-step audio generation does not strictly require massive paired audio-text datasets, addressing a major bottleneck in audio synthesis where such data is scarce.
• Quality vs. Speed: The framework successfully narrows the quality gap between one-step and multi-step systems, making real-time, resource-constrained applications more feasible.
• Semantic Controllability: The distilled model retains the semantic controllability of the teacher, allowing for word swapping, attention reweighting, and prompt refinement, indicating that the disentangled semantic space of the teacher is preserved in the one-step student.

The authors conclude that while the current model is limited to fixed-length (10s) segments and does not enforce linguistic control for speech, the proposed audio-free distillation paradigm offers a promising direction for fast, scalable, and controllable text-to-audio synthesis.