Mental Damage: Caption Poisoning Attacks on Retrieval-Augmented Text-to-Music Generation

Technical Summary: Mental Damage – Caption Poisoning Attacks on Retrieval-Augmented Text-to-Music Generation

Problem Statement
Retrieval-Augmented Generation (RAG) systems for Text-to-Music (TTM) aim to bridge the gap between high-level, underspecified user prompts (e.g., "calm music for studying") and the low-level acoustic details required for high-quality generation. These systems retrieve detailed music captions from a knowledge base to augment user prompts before synthesis. However, this architecture introduces a critical integrity dependency on the retrieval corpus. The paper identifies that an attacker can poison this music-text knowledge database by injecting a small number of crafted captions. The goal is to induce the system to retrieve malicious captions that bias the prompt augmentation process, steering the generated music toward an attacker-chosen target intent (e.g., horror music) while the user's original query remains benign and unchanged. Unlike direct model attacks, this method leaves the user prompt, retriever architecture, and generator parameters unaltered, making the attack difficult to detect from the user's perspective.

Methodology
The authors propose a dual-layer caption poisoning strategy designed to navigate the inherent trade-off between retrieval plausibility (the poisoned caption must be retrieved for the benign query) and steering strength (the caption must effectively redirect generation). The attack is formulated as optimizing a poisoned caption $c_p$ composed of two concatenated parts: $c_p = c_{anchor} \parallel c_{payload}$.

The strategy consists of three core components:

1. Anchor Preservation (Retrieval Plausibility):
   To ensure the poisoned caption is retrieved for a benign query, the attack preserves high-level semantic cues ($c_{anchor}$) that align with the user's original intent. For example, if the user asks for "study music," the poisoned caption retains phrases like "calm study music" or "relaxing background music." This ensures the caption remains a plausible match in the embedding space of the retriever.

2. High-level Functional Opposite Target Generation (Conflict Control):
   The attacker selects a target category ($t^*$) that is functionally opposite to the user's intent (e.g., shifting from "comfort/focus" to "unease/tension") but avoids excessive low-level acoustic conflict. The selection process maximizes functional opposition while minimizing descriptor-level incompatibility. For instance, "horror" is chosen over "rock" for a "calm study" query because horror can share low-level traits with calm music (e.g., slow tempo, sparse texture, sustained drones) while still achieving a drastic shift in user-perceived function. This balance prevents the poisoning from destabilizing the retrieval or generation process.

3. Low-level Semantic Similar Target Generation (Steering Strength):
   Instead of relying solely on high-level labels, the attack injects a payload ($c_{payload}$) of detailed, low-level acoustic descriptors derived from the target category. Examples include "distant echoing chimes," "hollow reverberant space," or "slow pulsing drone." Since TTM models rely heavily on these specific acoustic cues for conditioning, this payload effectively steers the downstream generation toward the attacker's target without breaking the semantic link established by the anchor.

Experimental Setup and Results
The authors evaluated the attack on a RAG-based TTM pipeline using the MusicCaps dataset as the knowledge base, the CLAP model as the retriever, and MusicGen (small) as the generator.

• Setup: The attacker injected a small number of poisoned captions into the database. The system was tested with benign queries (e.g., "give me calm background music for studying") and evaluated against a target category (e.g., "horror").
• Metrics: Performance was measured using CLAP similarity (cosine similarity between text and audio embeddings), Precision, Recall, and F1 scores for retrieval.
• Key Findings:
  • Steering Effectiveness: The attack successfully shifted the generated music toward the target category. The CLAP similarity between the poisoned generation and the target category increased substantially (roughly doubling from ~0.21–0.28 to ~0.41–0.48).
  • Stealth: The CLAP similarity between the generated music and the original user query remained largely unchanged (staying around 0.30), indicating that the attack did not degrade alignment with the user's prompt, making it difficult to detect based on query relevance alone.
  • Retrieval Success: The poisoned captions were successfully retrieved with high precision, recall, and F1 scores across different top-k settings, confirming the efficacy of the anchor preservation strategy.

Key Contributions

1. Identification of a New Attack Surface: The paper identifies caption poisoning in RAG-based TTM systems as a novel integrity threat, demonstrating that manipulating metadata in the retrieval corpus can redirect the high-level-to-low-level prompt translation process.
2. Dual-Level Poisoning Design: The authors propose a specific attack architecture combining Anchor Preservation, Function-Opposed Targeting, and Descriptor-Level Payload Injection to simultaneously satisfy retrieval constraints and maximize malicious steering.
3. Empirical Validation: The study demonstrates that this attack can effectively double the similarity of generated music to an attacker-chosen target intent while maintaining alignment with the benign user query, exposing a practical integrity risk in retrieval-augmented creative AI.

Significance and Claims
The paper claims to expose a practical integrity risk for retrieval-augmented creative AI systems. It argues that the security of these systems cannot rely solely on the robustness of the generative model; the integrity of the external retrieval corpus and the textual labels attached to audio are equally critical. The authors conclude that defenses must treat the retrieval corpus and caption metadata as part of the trusted base, necessitating stronger ingestion controls and robustness checks within the retrieval pipeline. The work highlights that in multimodal RAG systems, the attack surface extends beyond the generative model to include the embedding space and the external knowledge source.