Exploring How Audio Effects Alter Emotion with Foundation Models

Technical Summary: Exploring How Audio Effects Alter Emotion with Foundation Models

Problem Statement
While prior research has extensively mapped low-level audio features (e.g., tempo, pitch, timbre) and environmental conditions to emotional perception, the systematic impact of intentional audio effects (FX)—such as reverberation, distortion, modulation, and dynamic range processing—on emotional responses remains underexplored. Audio FX are central to contemporary music production for modulating aesthetic and emotional impact, yet no comprehensive study has systematically investigated how these effects, individually or in combination, alter emotional perception. Furthermore, the robustness of large-scale foundation models to these specific sound design manipulations is not well understood.

Methodology
The authors leveraged three state-of-the-art foundation models to analyze the relationship between audio FX and estimated emotion:

1. MERT-v1-330M: A music-specific model trained on 160,000 hours of audio using masked language modeling.
2. CLAP: A dual-encoder model (audio-text) trained on contrastive learning for zero-shot generalization.
3. Qwen2-Audio-7B: A large multimodal model trained on speech, music, and environmental audio.

The study utilized three datasets with diverse emotional annotations: EMOPIA (categorical emotions in pop piano), DEAM (continuous valence-arousal), and witheFlow (both dimensional and categorical tags).

To probe the models, the authors applied six common audio FX (Chorus, Distortion, Phaser, EQ, Delay, Reverb) at varying intensity levels (1–10) to input audio. They employed a "probing" approach where the foundation models served as frozen backbones, and shallow, interpretable classifiers (XGBRegressor for regression, XGBClassifier for classification) were trained on the extracted embeddings to predict emotional labels or dimensions.

The investigation proceeded through four experimental phases:

• Performance Impact: Measuring changes in model accuracy (MSE, F1-score) as FX intensity increased.
• Prediction Shifts: Analyzing changes in predicted emotional labels and dimensions (e.g., shifts in Anger vs. Calmness).
• Embedding Space Alterations: Using UMAP dimensionality reduction to visualize how audio FX reshape the internal latent representations of the models.
• Real-World Scenarios: Testing curated FX chains mimicking iconic artists (Pink Floyd, U2, Rage Against the Machine) to observe cumulative effects.

Key Results

• Performance Degradation: The application of audio effects generally led to a decrease in model performance across all tasks. Distortion and Phaser induced the most significant performance declines, with degradation becoming more pronounced as FX intensity increased.
• Emotional Prediction Shifts:
  • Distortion: Consistently increased predictions of Anger and decreased Calmness across all models.
  • Chorus and Delay: Introduced greater variability in predictions. Specifically, increasing Chorus boosted Calmness in CLAP and MERT, while increasing Delay raised Anger in CLAP and Qwen.
• Embedding Sensitivity:
  • CLAP: Exhibited large, structured displacements in embedding space, particularly for Delay, Chorus, and Distortion, indicating high sensitivity to timbral manipulations.
  • Qwen: Showed noticeable but less consistent trajectory shifts.
  • MERT: Remained relatively stable across all effect levels, suggesting robustness likely due to its music-specific training.
• Real-World Chains: Cumulative FX chains (e.g., heavy distortion or spatial effects) produced larger and more coherent shifts in embedding space compared to isolated effects. The trajectories were often unidirectional for distortion-heavy chains, reflecting uniform spectral shaping.

Significance and Contributions
The paper claims to advance the understanding of the perceptual impact of audio production practices by demonstrating that foundation models encode rich associations between musical structure, timbre, and affective meaning. By applying probing methods to these models, the authors uncover complex, nonlinear relationships between audio FX and estimated emotion.

The study establishes that:

1. Audio FX systematically alter emotional perception in ways that can be quantified via foundation model embeddings.
2. Different foundation models exhibit varying degrees of robustness to these manipulations, with MERT showing greater stability than CLAP or Qwen.
3. The magnitude and structure of latent-space shifts serve as metrics for quantifying the emotional impact of sound design, with real-world production chains generating the most pronounced effects.

These findings offer a framework for probing the emotional consequences of sound design techniques, with implications for music cognition, performance analysis, and affective computing. The authors note that future work will involve investigating simultaneous multi-effect applications, expanding to additional datasets, and comparing these findings against classical feature-based systems and human ground truth.