Technical Summary: AnchorSteer

Problem Statement

Controllable music editing aims to modify high-level attributes (e.g., instrument, genre) of a source audio while strictly preserving its rhythmic and melodic structures. Current diffusion-based approaches face a fundamental semantic-structural entanglement:

• Semantic Steering: Methods that rely solely on text guidance or latent steering (e.g., MusicMagus) often achieve strong attribute changes but degrade the source's temporal structure, effectively "re-synthesizing" the audio rather than editing it.
• Structural Anchoring: Methods that inject explicit structural conditions (e.g., MuseControlLite) preserve the musical skeleton well but often over-constrain the latent space, suppressing the model's ability to respond to semantic prompts and resulting in weak attribute transfer.

The core challenge is to enforce structural preservation without compromising semantic editability.

Methodology: AnchorSteer

The authors propose AnchorSteer, a framework that disentangles this tension by coupling structural anchoring with self-discovered semantic steering. The approach operates in two main phases:

1. Self-Discovered Concept Injection (Semantic Steering)

Instead of relying on curated datasets, the framework extracts interpretable, label-free concept vectors from a pre-trained text-to-music diffusion model using a self-supervised objective.

• Contrastive Prompting: A target prompt (e.g., "A piano music piece") and a base prompt (e.g., "A music piece") are used to generate reference samples.
• Optimization: The diffusion model parameters are frozen. A set of learnable concept injection modules ($F$) is optimized to reconstruct the target reference samples when the model is conditioned only on the generic base prompt.
• Mechanism: The modules learn to capture the "semantic gap" between the base and target prompts. Once trained, these modules act as portable, plug-and-play vectors that can be injected into the model's hidden states ($h$-space) to steer generation toward the target attribute.
• Variants:
  • Unconditioned Injection: Uses a static vector ($v_l$) added to hidden states, independent of the current state.
  • Conditioned Injection: Uses a lightweight network to dynamically compute the injection based on the input hidden state, offering stronger semantic transfer in constrained regimes.

2. Structure-Anchored Steering (Inference Pipeline)

During the editing inference stage, the framework synergistically combines two forces:

• Structure-Anchoring: The MuseControlLite adaptor is used to inject explicit structural conditions (melody, rhythm) extracted from the source audio. This enforces strict temporal alignment via RoPE-aware cross-attention, locking the musical skeleton.
• Semantic Steering: Simultaneously, the optimized concept injection modules are injected additively into the diffusion hidden states.
• Integration: The final hidden state is computed as the sum of the structurally guided state and the semantic modification ($\hat{h}_l = h_l + \lambda_{edit} f^*_l(h_l)$). This allows the model to traverse the semantic manifold while remaining tethered to the original structural scaffold.

Key Contributions

1. Label-Free Concept Discovery: The authors identify reusable editing directions in the hidden states of text-to-music diffusion models. They propose a self-supervised reconstruction objective to extract these as interpretable concept vectors without requiring curated labeled data.
2. AnchorSteer Framework: A structure-aware editing framework that explicitly couples self-discovered semantic steering with explicit structural anchoring to resolve the semantic-structural trade-off.
3. Dual Operating Points: The design of a plug-and-play injection module with two modes: unconditioned injection (favoring structural fidelity) and conditioned injection (enabling robust semantic transfer under heavy structural constraints).
4. Empirical Validation: Demonstration of state-of-the-art performance on the ZoME-Bench for instrument and genre editing, validated by both objective metrics and subjective listening tests.

Results

Experiments were conducted on the ZoME-Bench (instrument and genre change tasks) using the Stable Audio Open (SAO) backbone.

• Synergistic Design Validation:
  • Steering-only: Achieved high semantic shift (GAP) but poor structural fidelity (low Chroma, high LPAPS).
  • Anchoring-only: Preserved structure perfectly but failed to transfer attributes (low GAP).
  • AnchorSteer: Successfully balanced both, achieving a GAP of 0.198 (instrument) and 0.073 (genre) while maintaining high structural scores (Chroma ~0.47).
• Comparison with SOTA:
  • The Conditioned Injection variant achieved the highest GAP scores (0.279 for instrument, 0.136 for genre) among all baselines (including SDEdit, DDPM-Friendly, MusicMagus, and MuseControlLite).
  • It demonstrated superior ability to steer toward target semantics ($\Delta CLAP_T$) while suppressing source attributes ($\Delta CLAP_S$).
• Subjective Evaluation:
  • In a listening test with 28 participants, the Conditioned variant achieved the highest Target Attribute Match score (3.60/5) and the highest Audio Quality score (3.31/5).
  • While Content Consistency was slightly lower than the pure anchoring baseline (reflecting the necessary trade-off for semantic change), it remained competitive.
• Prompt Robustness: The Conditioned Injection variant showed remarkable robustness to text prompt conflicts, maintaining high performance even when the text prompt contradicted the injected concept, whereas the Unconditioned variant was more sensitive to such conflicts.

Significance and Claims

The paper claims to offer a novel framework that makes self-discovered semantic steering practically usable for diffusion-based music editing. By explicitly decoupling semantic manipulation from structural generation through the coupling of a structural adaptor and state-dependent concept injection, AnchorSteer enables significant semantic transformations with high-fidelity structural preservation.

The authors position the work not just as an editing tool, but as a contribution to the discovery, representation, and utilization of high-level concepts in generative models. They emphasize that the technical kernel lies in extracting interpretable, label-free concept vectors via self-supervision, which serve as portable semantic representations.

The paper modestly acknowledges limitations, noting that the current scope is limited to single-attribute editing (instrument/genre) and that the editable length is constrained by the backbone model (47 seconds). It does not claim to have solved multi-attribute joint editing or long-form continuity, identifying these as open challenges.