💻 computer science

Clean2FX: Label-conditioned modeling for clean-to-effect guitar audio transformations

The paper introduces Clean2FX, a label-conditioned framework for transforming clean electric guitar audio into various effects using U-Net models that outperform variational autoencoders in preserving musical content while accurately synthesizing target effects like distortion, delay, and reverb.

Original authors: Oliverio Bombicci Pontelli, Iran R. Roman

Published 2026-07-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Oliverio Bombicci Pontelli, Iran R. Roman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a plain, dry electric guitar sound—like a clean, unseasoned piece of toast. Now, imagine you want to turn that toast into something specific: a crunchy, distorted rock riff, a watery delay echo, or a huge, cavernous reverb. The paper "Clean2FX" is a recipe book for a digital chef (a computer model) that tries to do exactly this: take a clean guitar note and instantly cook up the perfect "flavor" of an effect, all while keeping the original melody and rhythm perfectly intact.

The researchers built their kitchen using a massive library of real guitar recordings called EGFxSet. They took single notes and chords, paired them with their "effected" twins, and taught four different types of digital chefs how to transform the clean sound into the target sound.

The Big Winner: The U-Net Chef
Out of the four chefs tested, two were "Variational Autoencoders" (VAEs) and two were "U-Nets." Think of the VAEs as artists who try to memorize the essence of a sound by compressing it into a tiny, blurry sketch before trying to redraw it. The U-Nets, on the other hand, are like master copyists who use a "skip connection" system. Imagine a U-Net as a chef who keeps the original ingredients (the clean guitar) on a conveyor belt right next to the cooking pot, adding the "spices" (the effects) layer by layer without ever losing the original texture.

The results were clear: the U-Net models were the clear winners. In fact, the paper notes that on average, the VAEs failed to beat a "do nothing" baseline (where the computer just plays the clean guitar without changing it). While one specific VAE variant did manage to improve the "RAT" distortion effect, they generally showed no positive improvement across the other effects, meaning they were, on average, no better than just leaving the guitar alone. The U-Nets, however, slashed the error rate by about 70.6% (using a log-magnitude loss) and improved the "perceptual quality" (a measure of how human ears judge the sound) by 31.8%.

The "Time" Problem: Distortion vs. Echo
Here is where it gets interesting. The paper found that the type of effect matters a lot.

Distortion (The Easy Win): Effects like the "RAT" or "Tube Screamer" are like heavy spices. They change the sound so drastically that the computer has a huge amount of "error" to fix. The U-Nets crushed this, improving the sound by over 80% in some cases.
Delay and Reverb (The Tricky Part): These are like adding a long echo or a room sound. The paper notes a strange split here: the U-Nets got the math right (reducing the spectral error by 78.6% for Spring Reverb), but the human ear didn't notice as much of a difference (only a 0.9% improvement in perceptual quality). It suggests that while the computer can calculate the echo perfectly, making it feel real to a human listener is still a bit of a challenge for these models.

Did the Chef Actually Listen?
A major fear in AI is "mode collapse," where a smart model gets lazy and just outputs the same thing no matter what you ask it. The researchers checked this by asking the model to generate all ten different effects for the same guitar note. They measured how different the outputs were from each other. The results showed a ratio of 1.416, which is well above zero. This suggests the model is actually listening to the "label" (the instruction) and changing its output accordingly, rather than just ignoring the instruction and spitting out the same sound every time.

The Real-World Test
Finally, the team took their best model (the U-Net with log-magnitude training) and tried it on real-world guitar performances that the model had never seen before. The results were "audible but weaker." The model could still add the flavor, but it wasn't as perfect as it was on the training data. The paper concludes that while the models work great on the specific data they were trained on, they are still learning how to handle the messy, unpredictable nature of real-life guitar playing.

In short, the paper proves that U-Net models are the best current tool for turning clean guitar into effects, beating out other methods by a wide margin, but it also suggests that while we are getting very good at the math of distortion, making digital echoes and reverbs sound truly natural to human ears is still a work in progress.

Technical Summary: CLEAN2FX

Problem Statement
The paper addresses the challenge of label-conditioned clean-to-effect transformation for electric guitar audio. The objective is to synthesize an effected signal from a clean guitar input given a specific target effect label, while strictly preserving the underlying musical content (pitch, rhythm, and playing technique). Unlike prior work often focused on single devices or parametric processors, this task requires a unified model capable of handling diverse effect families—ranging from broadband distortion to time-based modulation, delay, and reverb—without collapsing into a single transformation or ignoring the conditioning label.

Methodology
The authors propose Clean2FX, a framework comprising a custom dataset construction, four neural architecture comparisons, and a web-based demo.

Dataset Construction: Training data is derived from EGFxSet, a dataset of real hardware-processed guitar tones. To create musical sequences suitable for training, the authors programmatically assembled matched chords, melodies, and mixed event timelines. Crucially, the same event sequence is applied to both the clean and effected versions of each tone, ensuring the musical content remains constant while the effect serves as the sole variable. Pairs are normalized by the larger of the two signal maxima to preserve energy ratios.
Input Representation: Audio is represented as magnitude Short-Time Fourier Transform (STFT) spectrograms ($nfft=512$, $hop=128$, $16$ kHz). Two input representations are compared:
1. Linear shared-max normalized magnitude ( $x$ ).
2. Log-compressed magnitude ( $y = \ln(1 + 200x)$ ), inverted before metric computation.
  Phase is discarded during training and estimated via Griffin-Lim at inference.
Architectures: Four models are evaluated, all conditioned on learned effect embeddings:
1. VAE: A fully connected Variational Autoencoder with a 64-dimensional latent space, trained with standard MSE reconstruction and KL divergence.
2. ConvVAE: A convolutional VAE with a 512-dimensional deterministic latent space, using strided convolutions, BatchNorm, and ReLU, optimized with L1 loss on log-magnitudes.
3. U-Net (Linear): A five-level residual U-Net operating on linear spectrograms.
4. U-Net (Log): A five-level residual U-Net operating on log-magnitude spectrograms.
Conditioning Mechanism: The U-Net models utilize a sophisticated conditioning strategy. Effect labels are embedded into 64-dimensional vectors injected at three points: the bottleneck (via concatenation and convolution), decoder levels (via Feature-wise Linear Modulation, FiLM), and skip connections (via FiLM). Conditioning dropout (probability 0.15) is applied during training to prevent the model from ignoring the label. In contrast, VAEs condition only at the decoding stage via concatenation.
Loss Functions: U-Net variants optimize a composite spectral loss including Spectral Convergence (SC) and L1 terms. The log-magnitude variant uses SC + $10 \times L1_{log}$ . The linear variant uses SC + $10 \times L1_{lin} + 5 \times L1_{log}$ to penalize low-energy errors. A per-effect weighting scheme upweights effects with naturally small spectral differences to ensure gradient signals do not vanish.

Key Results
Evaluation is performed on held-out paired samples using Mean Squared Error (MSE) on linear spectrograms and Fréchet Audio Distance (FAD).

Model Performance: The U-Net (Log) model achieved the best performance, improving MSE by 70.6% and FAD by 31.8% over the unprocessed input baseline. The U-Net (Linear) followed closely with a 69.4% MSE improvement and 23.0% FAD improvement.
VAE Limitations: Both VAE and ConvVAE baselines failed to outperform the "copy-input" baseline on average, showing no positive mean per-effect improvement. The authors attribute this to the latent bottleneck compressing fine harmonic details, leading to blurred reconstructions.
Effect-Specific Trends:
- Distortion: These effects showed the most significant improvements, as they drastically alter the signal, leaving substantial error for the model to correct.
- Time-Based Effects (Delay/Reverb): These exhibited a divergence between metrics. While MSE improvements were strong (e.g., Spring Reverb +78.6%), FAD gains were notably weaker (e.g., Spring Reverb +0.9%). This suggests that while spectral error is reduced, perceptual quality improvements are more challenging for time-based transformations.
Conditioning Sensitivity: A diagnostic test measuring the pairwise distance between outputs for different labels confirmed that the best model responds to the conditioning labels (mean ratio of 1.416) rather than collapsing to a single transformation.

Significance and Claims
The paper presents Clean2FX as both a technical study and a public demonstration. Its primary contributions are:

Controlled Evaluation: The construction of a dataset where musical content is held fixed across clean/effected pairs allows for a rigorous, controlled comparison of transformation capabilities across multiple effect families.
Architectural Insight: The results demonstrate that U-Net architectures with skip connections significantly outperform VAEs for this task, likely due to the preservation of harmonic structure across resolutions.
Representation Matters: The study highlights that log-magnitude representations yield superior perceptual quality (FAD) compared to linear representations, despite similar MSE performance, particularly for effects involving low-amplitude details like reverb tails.
Practical Demonstration: The accompanying demo website applies the trained models to real-world guitar performances outside the training distribution (Guitar-TECHS), illustrating that while the models produce audible transformations, performance degrades on out-of-distribution data compared to the controlled EGFxSet environment.

The authors conclude that while the models successfully learn label-conditioned transformations for hardware-derived effects, the gap between spectral error reduction and perceptual quality remains a challenge, particularly for time-based effects.

Technical Summary: CLEAN2FX

More like this