A Text-Steerable Instrument for Sketching Procedural Soundscapes via Language Models

Technical Summary: A Text-Steerable Instrument for Sketching Procedural Soundscapes via Language Models

Problem Statement
Current neural text-to-audio systems (e.g., MusicGen, MusicLM, Stable Audio) generate high-quality monolithic waveforms but suffer from significant limitations in live performance contexts. These systems typically exhibit high generation latency (often requiring GPU acceleration), produce non-interpretable raw audio that cannot be easily edited post-generation, and lack "steerability"—the ability for a performer to make continuous, predictable adjustments to the sound without stopping the stream. This creates a "tweakability problem" where the performer cannot treat text as a real-time control surface.

Methodology
The paper presents a software development kit (SDK) and instrument architecture that decouples text instruction resolution from audio emission. Instead of generating raw audio, the system uses language models (LLMs) or retrieval mechanisms to output interpretable, symbolic synthesizer configurations.

• Architecture (The Live Generator): The core innovation is an asynchronous streaming interface. A performer yields instructions (text prompts, absolute configurations, or relative parameter nudges) via a Python generator. The system continuously emits audio from the current configuration while the next configuration resolves in the background (via an LLM API, a local model, or embedding retrieval). Once resolved, the system seamlessly crossfades into the new configuration, ensuring uninterrupted audio even if resolution takes 5–12 seconds.
• Configuration Schema: The system utilizes a 34-field categorical schema divided into five groups: global parameters (tempo, brightness, space, etc.), six orchestration layers (bass, pad, melody, etc.), spatial/texture, melody generation, and harmony. The schema is designed so that most valid combinations produce musically coherent results. It supports both absolute assignments and relative stepping (e.g., Step(-2) on brightness).
• Backends: The system supports three interchangeable backends that emit the same schema:
  1. Fast Mode (Default): Uses embedding retrieval against a pre-computed map of ~10,500 curated scene descriptions. This runs on CPU with sub-second latency and no network dependency.
  2. External Mode: Routes prompts to hosted LLMs (e.g., via API). It handles network latency by keeping the current audio playing during resolution.
  3. Local Expressive Mode: Runs a fine-tuned 270M parameter model on-device (experimental).
• Dataset Construction: The retrieval map was constructed by distilling scene descriptions from the Common Pile corpus, generating candidate configurations with an LLM, scoring them using LAION-CLAP (a contrastive language–audio model) for text–audio semantic similarity, and selecting the best matches.

Key Contributions

1. Live Generator Architecture: A runtime that hides resolution latency behind continuous audio, reframing text-to-music from a "one-shot" generation task into an ongoing, performable stream.
2. Mapping Design: A 34-field categorical configuration schema where valid combinations are biased toward musical coherence, enabling fine-grained performer control without requiring knowledge of digital signal processing (DSP).
3. Open-Source Release: The release of the SDK, the dataset artifacts, the fine-tuned model, and the audiovisual performance interface.

Results and Evaluation

• Performance: In a benchmark of 200 held-out prompts, the embedding retrieval backend resolved all prompts successfully with a total latency of ~1.2 seconds (including synthesis). The external LLM backend resolved 178/200 prompts (89% success rate) with ~5.6 seconds of generation time; failures gracefully fell back to the fast backend to maintain continuity.
• Semantic Alignment: A technical evaluation using LAION-CLAP on held-out prompts showed that retrieval-based configurations scored higher on text–audio semantic similarity than random valid configurations. The authors note this metric serves as a proxy for system reliability and alignment, not a measure of perceptual quality, and that the retrieval map was constructed using this same metric.
• User Feedback: Informal testing with five lay participants indicated that the output was perceived as "musically intentional" rather than random, supporting the schema's design goal. Participants noted the system excels at gradual scenic evolution but struggles with sharp stylistic pivots or specific timbral requests (e.g., "upright bass").

Significance and Claims
The paper claims to address the tension between the high quality of neural audio and the need for low-latency, reproducible, and steerable control in live performance. By generating symbolic parameters rather than raw audio, the system offers:

• Immediacy and Continuity: Audio never stops during configuration resolution.
• Reproducibility: Given a configuration and seed, the output is identical, addressing concerns about the longevity and documentation of deep generative models in NIME (New Interfaces for Musical Expression) practice.
• Resource Efficiency: The default CPU-only backend amortizes LLM costs into a one-time dataset construction step, avoiding the need for heavy inference at runtime.
• Extensibility: The open-source nature and shared vocabulary of the parameter space allow other developers to extend the system with new style sets or synthesizer backends.

The authors position the instrument as a tool for sketching procedural soundscapes, particularly effective for ambient, cinematic, and loop-adjacent contexts where texture and harmonic color are prioritized over authored melody or strict genre imitation.