SAKE: Towards Editing Auditory Attribute Knowledge of Large Audio-Language Models

1. Problem Statement

While Knowledge Editing (KE) has been extensively studied for Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) to update factual knowledge (e.g., "Paris is the capital of France") without full retraining, Large Audio-Language Models (LALMs) remain largely unexplored in this domain.

The core challenge addressed by this paper is that auditory knowledge is fundamentally different from factual or visual knowledge:

• Nature of Knowledge: Auditory attributes (e.g., speaker emotion, gender, animal sounds) are perceptual abstractions rather than discrete facts. They rely on high-level generalization over signals with significant intra-class variability (e.g., different speakers saying "happy" in different ways).
• The Gap: Existing editing methods designed for factual triplets (Subject-Relation-Object) may not generalize to modifying these abstract perceptual concepts. Furthermore, it is unclear if editing one auditory attribute (e.g., changing a frog's sound to a dog's) correctly propagates to related reasoning (e.g., changing the animal's diet from "insectivore" to "omnivore").
• Goal: To evaluate whether current KE methods can effectively edit, generalize, and preserve knowledge in LALMs when dealing with abstract auditory attributes.

2. Methodology: The SAKE Benchmark

The authors introduce SAKE (Speech and Audio Attribute Knowledge Editing Benchmark), the first comprehensive framework for evaluating auditory attribute editing.

A. Problem Formulation

Given an LALM $f$ with parameters $\theta$ and an editing dataset $D_{edit} = \{(a_e, x_e, y_e)\}$ (where $a_e$ is audio, $x_e$ is text, and $y_e$ is the target label), the goal is to update $\theta$ to $\theta'$ such that the model generates $y_e$ for the input, while preserving unrelated knowledge.

B. Evaluation Dimensions

SAKE evaluates editing performance across four dimensions, adapted from prior work but tailored for audio:

1. Reliability: Does the model correctly generate the new target for the specific editing instance?
2. Generality: Does the edit generalize to the "equivalent neighborhood"?
   • Type 1: Textual paraphrases.
   • Type 2: Acoustically similar audio (same label, different recording).
   • Type 3: Both audio and text variations.
3. Locality: Does the edit preserve unrelated knowledge?
   • Audio Locality: Preserving other attributes (e.g., editing emotion shouldn't change gender recognition) and other labels within the same attribute.
   • Text Locality: Preserving text-only capabilities.
4. Portability: Does the edit propagate to connected knowledge? (e.g., If "frog" becomes "dog," does the model update the animal's physical traits and diet?)

C. Experimental Setup

• Models: Three diverse LALMs were tested: DeSTA2.5-Audio, Qwen2-Audio, and Audio Flamingo 3.
• Attributes: Speaker gender, emotion, spoken language, and animal sounds.
• Methods: Eight representative editing methods were evaluated:
  • Fine-tuning: FT (LLM backbone) and FT (Modality Connector).
  • Hypernetworks: KE (Knowledge Editor), MEND.
  • Optimization: UnKE.
  • In-Context Learning: I-IKE, IE-IKE.
  • Memory-based: WISE.
• Settings: Evaluated under Single Editing and Sequential Editing (applying 10 edits in a row to test catastrophic forgetting and model degeneration).
• Evaluation: Used an LLM-as-a-judge (GPT-5 mini) with human verification (98.1% agreement).

3. Key Results

A. Single Editing Performance

• Reliability: Parameter-updating methods (FT, KE, MEND, WISE) achieved near-perfect reliability ($\ge 95\%$). In-Context Learning (IKE) methods performed poorly, likely due to LALMs' limited ability to leverage few-shot examples for audio.
• Generality: A significant gap exists between reliability and generality. While models learn the specific edit, they struggle to generalize to acoustically similar inputs (Type 2) or combined variations (Type 3).
  • FT (Audio) (editing the modality connector) showed better balance but struggled with emotion generalization.
  • FT (LLM) (editing the backbone) achieved high generality but at the cost of locality.
• Locality:
  • Intra-attribute entanglement: Preserving knowledge within the same attribute (e.g., editing "happy" to "sad" without affecting "fear") was the hardest task (Type 2 Audio Locality).
  • FT (Audio) maintained perfect Text Locality (since it doesn't touch the LLM backbone), whereas FT (LLM) degraded text capabilities significantly.
• Portability: Most methods failed to propagate edits to connected knowledge (e.g., changing the animal sound did not update the animal's diet). FT (Audio) showed the most balanced portability, suggesting the modality connector is a better locus for editing perceptual concepts.

B. Sequential Editing Performance

• Forgetting: Most methods suffered severe forgetting as the "edit gap" increased.
• Degeneration: Methods like MEND and KE exhibited catastrophic model collapse under sequential edits, producing repetitive or nonsensical outputs (e.g., "dogdogdog...").
• Stability: FT (Audio) and I-IKE showed greater robustness against degeneration, though I-IKE had low initial reliability.

4. Key Contributions

1. SAKE Benchmark: The first benchmark specifically designed for editing perceptual auditory attribute knowledge in LALMs, covering four critical dimensions (Reliability, Generality, Locality, Portability).
2. Empirical Insights:
   • Identified that intra-attribute entanglement (acoustic and semantic overlap within a single attribute) is a major bottleneck for current editing methods.
   • Demonstrated that editing the modality connector (FT Audio) is a more robust and balanced baseline than editing the LLM backbone for auditory tasks, preserving text capabilities and offering better portability.
   • Revealed that sequential editing often leads to model degeneration and severe forgetting, a limitation not fully addressed by current state-of-the-art methods.
3. Future Direction: Highlighted the need for auditory-specific editing techniques that can handle abstract perceptual generalization and prevent intra-attribute interference.

5. Significance

This work bridges a critical gap in the multimodal AI landscape. As LALMs are increasingly deployed in real-world applications (voice assistants, accessibility tools, content moderation), the ability to correct errors, mitigate biases, and personalize these models without retraining is essential.

SAKE reveals that methods successful in text or vision domains do not directly transfer to audio due to the unique nature of perceptual knowledge. By establishing a rigorous evaluation framework and identifying the failure modes of current techniques (specifically regarding generalization and sequential stability), this paper provides a foundational roadmap for developing the next generation of auditory-specific knowledge editing algorithms.