Go witheFlow: Real-time Emotion Driven Audio Effects Modulation

Technical Summary of "Go witheFlow: Real-time Emotion Driven Audio Effects Modulation"

Problem Statement
Current AI applications in music predominantly focus on autonomous generation, classification, or recommendation, often positioning AI as a replacement for human creativity rather than a collaborative tool. Furthermore, traditional music technology processes audio signals in isolation, overlooking the performer's real-time emotional and physiological state. There is a lack of systems that utilize lightweight, local AI to provide instant, low-latency feedback to live performers, bridging the gap between a musician's internal state and their sonic output without introducing the latency or computational overhead of cloud-based solutions.

Methodology
The authors propose witheFlow, a proof-of-concept system designed to enhance real-time music performance by automatically modulating audio effects based on a fusion of biosignals and audio analysis. The system is implemented in Python, runs locally on a laptop, and communicates via MIDI protocol messages with a Digital Audio Workstation (DAW).

The architecture consists of three primary components:

1. Biosignal Feature Extraction:

   • Sensors: The system utilizes commercial-grade Electroencephalography (EEG) and Electrocardiography (ECG) sensors.
   • EEG Processing: Signals are sampled at 250 Hz from electrodes at O1, O2, T3, and T4. Power is extracted from Alpha (8–13 Hz) and Beta (13–30 Hz) bands over 4-second sliding windows. Two metrics are derived: Attention (Beta Power / (Alpha + Beta)) and Relaxation (Alpha Power / (Alpha + Beta)).
   • ECG Processing: Signals are sampled at 1000 Hz. The Baevsky Stress Index (SI) is computed using a sliding window of 15 seconds (analyzing 30 windows of 0.5s). This index quantifies stress by reflecting the balance between sympathetic and parasympathetic nervous system activity.
   • Robustness: The system includes artifact detection. If persistent artifacts (e.g., poor contact) are detected, the affected device is deactivated. For transient EEG artifacts, features are extracted from remaining electrodes; if all fail, values are imputed from previous time steps.

2. Audio Emotion Regressor:

   • Model: A PANNs CNN10 architecture, trained on the DEAM dataset, is used. The final classification layer is replaced with a linear regressor to output two values: Valence and Arousal.
   • Input: The "dry" audio (unprocessed) is down-sampled to 30 kHz and processed in 5-second windows.
   • Limitation: The authors note that hyperparameter tuning is minimal and that current datasets typically contain full productions rather than solo performances, which is a target for future dataset development.

3. Rule-Based Mixing Logic:

   • Mechanism: The system routes the dry audio through multiple parallel effect chains in the DAW. A mixing logic module dynamically adjusts the gain of each channel (including the dry signal) based on a combination of the performer's physiological state (Stress, Attention) and the audio's affective state (Valence, Arousal).
   • Configuration: Logic is defined via customizable YAML files containing rules (e.g., stress < x < attention). These rules trigger Python functions that boost or suppress specific effect channels.
   • Strategy: The current rulesets aim to align the output mix with the performer's intent. For example, under High Stress/High Attention, the system boosts effects that are "far" from the dry signal in Valence-Arousal (VA) space to create a contrasting sonic environment. Under Low Stress/Low Attention, it boosts effects close to the dry signal with lower arousal.
   • Control: Performers can modulate the strength of these rules or override them entirely via MIDI foot pedals.

Key Contributions

• System Architecture: The introduction of witheFlow, a lightweight, open-source system that integrates biosignals and audio analysis for real-time audio effect modulation without cloud dependency.
• Hybrid AI Approach: A combination of machine learning (for emotion regression and biosignal feature extraction) and transparent, rule-based logic (for mixing decisions) to ensure interpretability and controllability.
• Local Execution: A design that prioritizes low latency, data privacy, and portability, allowing the system to run entirely on a local machine.
• Robustness Mechanisms: Implementation of artifact detection and adaptive rule-switching to maintain functionality during sensor signal degradation.

Results and Evaluation
The paper presents the system in a proof-of-concept phase.

• User Feedback: The system was co-developed and tested with multiple musicians. Participants reported a generally positive experience, particularly during improvisational sessions, noting that the system's responsiveness to their internal state opened new expressive possibilities.
• Technical Status: The system successfully routes audio, processes biosignals, and applies rule-based gain adjustments in real-time. However, the authors explicitly state that significant hyperparameter tuning has not yet been performed and that a dedicated multi-instrumental dataset for solo performances is currently under development to improve the audio regressor.

Significance and Claims
The authors position witheFlow not as an autonomous composer, but as a tool that preserves human creative agency while leveraging AI to handle technical processing.

• Human-Centric Collaboration: The system aims to create a "liberated environment" where AI acts as an enhancing tool rather than a competitor, allowing performers to focus on self-expression.
• Bio-Awareness: By creating a direct connection between internal emotional experience and sonic output, the system facilitates "bio-awareness" and mind-body integration for musicians.
• Future Directions: The paper outlines several research directions, including the creation of formal evaluation frameworks, the development of larger datasets containing biosignals and solo performances, and the exploration of hybrid architectures where local interpretable models handle real-time interaction while cloud-based systems provide higher-level insights.
• Ethical Stance: The authors emphasize the importance of privacy (via local processing), informed consent, and the performer's ability to override the system to ensure psychological safety and maintain authorship of the performance.