ExSampling: a system for the real-time ensemble performance of field-recorded environmental sounds

The paper proposes ExSampling, an integrated system combining a recording application and a Deep Learning environment to enable the real-time ensemble performance of field-recorded environmental sounds through automated sound mapping to Ableton Live tracks.

The Gist

Imagine you are at a concert, but instead of the musician just playing a guitar or a synthesizer, they are conducting a symphony made entirely of the sounds happening right now all over the world.

That is the core idea behind ExSampling, a system created by researchers at Keio University. Here is a simple breakdown of how it works, using some everyday analogies.

The Problem: The "Library" Bottleneck

Traditionally, if a musician wants to use the sound of a rainstorm or a busy street in their song, they have to go out, record it, come back, listen to hours of tape, find the perfect 5-second clip, and then manually teach their computer how to play it.

It's like trying to cook a gourmet meal, but you have to spend three hours just chopping vegetables before you can even turn on the stove. By the time you are ready to cook, the inspiration is gone, and the "freshness" of the moment is lost.

The Solution: The "Magic Translator"

ExSampling is like a real-time magic translator that turns the chaos of the outside world into a musical instrument instantly. It connects three different groups of people:

1. The Field Recorders: Regular people with smartphones or laptops anywhere in the world.
2. The Performer: A musician on stage (or online) playing a show.
3. The AI Brain: A computer program that does the heavy lifting.

How It Works (The Three Steps)

1. The "Ears" (The Web Recorder)
Imagine a website that acts like a universal microphone. Anyone can visit this site, press a button, and record whatever sound is around them—a barking dog, a train whistle, or wind in the trees.

• The Magic: As soon as they hit record, that sound is instantly sent over the internet to the musician's computer. No waiting, no downloading.

2. The "Brain" (The Deep Learning AI)
This is the coolest part. The musician's computer has a "brain" (a Neural Network) that listens to the incoming sound and instantly guesses what it is.

• The Analogy: Think of it like a very fast, very smart librarian. You hand them a book (the sound), and they immediately shout, "This is a bird!" or "This is a car!"
• The system then converts that sound into a visual map (a spectrogram) and asks the AI, "What is this?" The AI answers in milliseconds.

3. The "Conductor" (The Music Software)
Once the AI knows what the sound is, it automatically assigns it to a musical instrument.

• If the AI hears a bird chirping, it might assign that sound to a flute track.
• If it hears a car honking, it might assign it to a drum track.
• If it hears wind, it might assign it to a synthesizer.

The musician doesn't have to do any of this sorting. They just play their MIDI keyboard, and the system instantly swaps in the fresh, real-world sound that matches the note they are playing.

The Experience: A Global Jam Session

Imagine a performer in Tokyo playing a drum solo.

• A student in London records the sound of a subway train. The system hears "train," maps it to a "bass drum," and the London train sound plays in the Tokyo concert.
• A hiker in the Alps records a rushing river. The system hears "water," maps it to a "hi-hat," and the river sound joins the beat.

The performer gets a notification on their screen showing a map of where the sounds came from. They can see, "Oh, the bass drum is coming from London right now!"

Why This Matters

• Serendipity (Happy Accidents): The musician doesn't know exactly what sounds will arrive. Maybe the AI mistakes a cat meow for a violin. That "mistake" might create a weird, beautiful sound that the musician loves. It adds an element of surprise and tension to the show.
• Democratizing Music: You don't need to be a professional sound engineer to contribute. If you have a phone and a quiet moment, you can be part of a live concert.
• Preserving the Moment: It captures the "vibe" of a specific place and time and turns it into music instantly, keeping the performance feeling alive and connected to the real world.

The Future

The researchers admit the system isn't perfect yet. Sometimes the AI guesses wrong, and the musician can't always choose to reject a sound they don't like. But they plan to add features like sending photos or videos along with the sound, so the audience can see exactly where the "drum beat" was recorded.

In short: ExSampling turns the whole world into a musical instrument, allowing a performer to play a live concert using the sounds of the world as they happen, with a smart computer acting as the bridge between the noise of reality and the harmony of music.

Technical Summary

Here is a detailed technical summary of the paper "ExSampling: a system for the real-time ensemble performance of field-recorded environmental sounds."

1. Problem Statement

The paper addresses a significant bottleneck in Musique Concrète and computer music performance: the latency between field recording and live performance.

• The Issue: Traditionally, artists must manually listen to, select, equalize, and map environmental sound samples to musical instruments (MIDI/samplers) before a performance. This process is time-consuming, limits the number of usable samples, and destroys the "real-time" nature of the performance.
• The Gap: While Deep Learning has been applied to music genre recognition and pitch detection, there was no existing system that utilized machine learning to automate the selection and mapping of field-recorded environmental sounds for real-time musical expression.

2. Methodology

The authors propose ExSampling, an integrated system comprising three distinct modules that bridge web-based recording, deep learning inference, and digital audio workstation (DAW) performance.

A. System Architecture

1. Web-Based Recorder (Frontend):

   • Technology: Built using p5.js and the Web Audio API (via recorder.js).
   • Function: Allows remote participants to record environmental sounds via their device microphones.
   • Connectivity: Uses ngrok for network port forwarding, enabling global participants to send audio data to the performer's local server via a QR code interface.
   • Feedback: Displays the recognized sound class and geolocation (via MapBox API) to the recorder, fostering a sense of contribution.

2. Deep Learning Inference Server (Backend):

   • Technology: Python-based server using TensorFlow and Librosa.
   • Preprocessing: Incoming audio is trimmed (removing silence below 20dB), segmented into 5-second increments, and converted into spectrograms (3-channel image matrices).
   • Model: Utilizes MobileNetV2, a lightweight Convolutional Neural Network (CNN), chosen for low-latency inference on local hardware.
   • Training Data: Trained on a Kaggle dataset containing 41 annotated classes of environmental sounds.
   • Output: The server outputs the sound file path, estimated class, detected pitch, and location data.

3. Performance Interface (DAW Integration):

   • Technology: Max/MSP (specifically Max for Live within Ableton Live).
   • Communication: Receives data from the Python server via OSC (Open Sound Control).
   • Function: Automatically maps the incoming sound file to a specific MIDI track based on the classification result. It triggers the sample playback in real-time.

B. Sound Mapping Logic

• Pitch Detection: The system uses Librosa to detect the dominant pitch of the original sample.
• Instrument Assignment:
  • Percussive sounds (e.g., footsteps, rain) are mapped to drum kits (hi-hats, snares) and played at their original pitch.
  • Melodic sounds (e.g., wind, birds) are mapped to melodic instruments (piano, synths) and are pitch-shifted to match the MIDI note input.
• Configuration: Mappings can be customized via Python configuration files or Max for Live device options.

3. Key Contributions

• First-of-its-Kind Application: This is the first research to propose applying machine learning specifically for the real-time selection and mapping of field-recorded environmental sounds in musical performance.
• Automated Workflow: The system eliminates the manual "listening and sorting" phase, allowing performers to use fresh, live field recordings instantly.
• New Interaction Model: It creates a tripartite interaction between Recording Participants (who capture sounds), the System (which classifies and maps them), and the Performer (who integrates them into the ensemble).
• Serendipity in Performance: The system introduces a controlled element of unpredictability. Because the performer does not know exactly what sound will arrive next, the automatic (and occasionally imperfect) classification creates "unconventional sound mappings," generating unique musical tension and expression.

4. Results and Performance Use Case

• Live Demonstration: The system was successfully demonstrated in a performance setting where remote participants recorded sounds via a web interface.
• Real-Time Latency: The integration of MobileNetV2 and local server processing allowed for near-instantaneous classification and sample triggering.
• User Experience:
  • Performers could treat environmental sounds as both rhythmic and melodic elements, mixing them with pre-recorded tracks or live instruments.
  • Participants received immediate feedback on how their recordings were classified and used, enhancing their engagement and sense of contribution to the "soundscape."

5. Significance and Future Work

• Artistic Impact: ExSampling democratizes the creation of concrete music, allowing non-musicians to contribute to live performances and encouraging musicians to incorporate real-world environmental contexts into their work.
• Future Improvements:
  • Visual Integration: Expanding the system to transmit video and photos alongside audio for Visual Jockeys (VJs) to create synchronized visual performances.
  • Human-in-the-Loop: Developing a Max for Live interface that allows performers to listen to incoming samples and manually accept or reject them before triggering.
  • Dataset Expansion: Improving the classification model by training on larger, more diverse datasets specifically curated for musical expression to increase the variety of available instruments.
  • System Integration: Merging all components (recorder, server, DAW) into a single, unified software package to improve usability.

In conclusion, ExSampling successfully bridges the gap between field recording and live performance, transforming the static archive of environmental sounds into a dynamic, real-time, collaborative musical instrument.