Genre Controlled Music Generation via Activation Steering

Technical Summary: Genre Controlled Music Generation via Activation Steering

Problem Statement
Computational music generation is increasingly moving toward non-conventional styles, creating a demand for methods that allow precise, controllable blending of diverse musical elements. While prior conditional music generation works have utilized text-based prompts (e.g., genre or mood descriptions) or fine-tuning for instrument and chord control, these approaches often require significant computational resources or lack fine-grained control. Existing inference-time intervention techniques have largely been applied to discrete attributes or language models, leaving a gap in lightweight, effective methods for controlling continuous, non-binary concepts like musical genres in autoregressive transformers.

Methodology
The authors propose a method for fine-grained genre control using inference-time interventions on the autoregressive generative transformer, MusicGen. The approach is grounded in mechanistic interpretability and activation engineering, drawing inspiration from techniques used to interpret Large Language Models (LLMs).

1. Concept Identification and Linear Probing:

   • The authors analyze the residual stream activations and attention layer outputs of MusicGen to identify where musical concepts (specifically genres: rock, classical, electronic, and jazz) are linearly represented.
   • They apply $k$-means clustering to activations across layers and evaluate the Adjusted Rand Index (ARI) and Normalized Mutual Information (NMI) to determine the layer where genre information is most distinct.
   • Linear Probes: A linear classifier (probe) is trained on the residual stream activations ($h^l_a$) of specific layers to predict genre labels. The probe learns a mapping $\hat{y} = W^\top h^l_a + b$. The layer yielding the highest probe accuracy is identified as the optimal location for the concept's linear representation.

2. Activation Steering:

   • Once the optimal layer and the steering vector (derived from the probe weights) are identified, the authors perform inference-time intervention.
   • A unit-magnitude steering vector $r^l$ is added to the residual stream activations before the target layer. The intervention is scaled by a hyperparameter $\alpha$ (searched in the range $[0.3, 1.0]$) and normalized:
     $$h^l_i \leftarrow h^l_i + \frac{\alpha r^l}{1 + \alpha}$$
   • This intervention is applied selectively to the conditional residual stream, allowing the model to steer the generation toward a target genre while preserving other features like tempo and melodic lines, which the authors hypothesize are represented in different layers.

3. Experimental Setup:

   • The experiments utilize the musicgen-small model and a balanced dataset of 10-second audio segments from the lewtun/music_genres dataset (900 samples per class for rock, classical, electronic, and jazz).
   • Transitions were tested in pairs (e.g., Classical $\to$ Electronic, Rock $\to$ Classical) and compared against a baseline where transitions were achieved solely through text prompting.

Key Contributions
The paper outlines three primary contributions:

1. User-Controllable Steering: A method enabling users to steer MusicGen activations to achieve specific genre control without retraining the model.
2. Extension to Continuous Concepts: The extension of compute-efficient activation steering techniques, previously used for discrete attributes, to continuous, non-binary concepts such as musical genres.
3. Validation: A comprehensive evaluation using both quantitative metrics (CLAP-based) and human perceptual studies (2AFC listening tests) to validate that activation steering enhances genre control while maintaining musical coherence.

Results
The study evaluated the method through two lenses:

• Quantitative (CLAP Scores): Using the clap-htsat-unfused model for zero-shot audio classification, the authors measured the similarity of generated audio to target genre labels. Results showed that steering consistently produced larger positive shifts in CLAP scores compared to text prompting. For example, in the "Classical $\to$ Electronic" transition, the steered method yielded a $\Delta$CLAP of $+0.2057$, compared to $+0.1276$ for prompting.
• Qualitative (Human Listening Study): A 2-Alternative Forced Choice (2AFC) study involving 24 participants (7 trained musicians and 17 non-musicians) revealed a consistent preference for steered samples over prompted baselines across all genre transitions. Trained musicians showed high accuracy in preferring steered samples (e.g., 80.5% for Rock $\to$ Classical, $p < 0.0001$), and the general audience also showed significant preference in most cases.

Significance and Claims
The paper claims that activation steering offers a lightweight, interpretable, and effective alternative to heavy fine-tuning or text-only conditioning for controllable music generation. By framing activation steering as a human-controllable interaction, the work demonstrates that interpretable model behaviors can empower co-creative workflows.

The authors modestly note that while CLAP scores serve as a useful proxy for genre alignment, they likely capture correlated musical attributes (timbre, tempo, rhythm) rather than explicit genre semantics. The steering strength is presented as an interactive parameter for users rather than a fixed hyperparameter. The work concludes that this approach validates the potential of inference-time interventions for manipulating musical features even in the absence of clearly contrasting samples, paving the way for more generalizable and interpretable strategies in controllable music generation.