🤖 machine learning

Generative Modeling of Bach-Style Symbolic Music: A Comparative Study of Autoregressive, Latent-Variable, and Adversarial Approaches

This paper compares autoregressive, latent-variable, and adversarial models for generating Bach-style symbolic music, finding that attention-based LSTMs produce the most coherent compositions while vector-quantized VAEs offer structured outputs and adversarial networks struggle with reliable generalization.

Original authors: Dezhi Yu, Kyuil Lee, Yongkang Huang

Published 2026-06-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dezhi Yu, Kyuil Lee, Yongkang Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library of sheet music written by Johann Sebastian Bach, the master of complex, elegant piano pieces from the 1700s. Your goal is to build a computer program that can listen to this library and then write its own new Bach-style songs from scratch.

This paper is a report card on three different "student" computer models the researchers tried to teach this task. They wanted to see which student could best mimic Bach's unique voice, learn the hidden rules of his music, and create something that sounds like a real composition rather than random noise.

Here is how the three students performed, explained with simple analogies:

1. The "Step-by-Step" Student (Autoregressive LSTM with Attention)

The Approach: Imagine a student who writes a story one word at a time. After writing "The," they think about what word usually comes next. They keep doing this, building the sentence slowly. This model does the same with music notes. It looks at the notes it just wrote and predicts the next one.
The Secret Weapon: The researchers gave this student a special pair of "highlighter glasses" (called Attention). Sometimes, when writing a long story, you need to remember the very first sentence to make sense of the last one. These glasses let the model look back at important notes from earlier in the song to decide what to play next.
The Result: This was the star student. It produced the most coherent, Bach-like music. It understood the flow, the chords, and the melody. It was like a student who not only memorized the rules but understood the feeling of the music.

2. The "Compressed Summary" Students (VAEs and VQ-VAEs)

The Approach: Imagine trying to learn Bach's music by first compressing a whole symphony into a single, tiny summary note, and then trying to expand that summary back into a full song. This is what Variational Autoencoders (VAEs) try to do. They try to find a hidden "secret code" (a latent space) that represents the music.
The Problem: Most of these students suffered from "Posterior Collapse." Think of this as a student who gets so scared of guessing the wrong summary code that they just ignore it entirely. They end up writing music that sounds okay (like jazz or random scales) but lacks the specific structure of Bach. They forgot the "secret code" and just guessed.
The Fix: One student tried a trick called Vector Quantization (VQ-VAE). Instead of trying to compress the whole song into one fuzzy cloud, this student broke the music into small, distinct "blocks" (like 4-note chunks) and learned a dictionary of these blocks.
The Result: This student did better than the others in this category. It learned to recognize specific patterns (like little scales going up and down) and produced music that sounded a bit more like the Baroque era, though it still wasn't as perfect as the Step-by-Step student.

3. The "Art Critic vs. The Artist" (Generative Adversarial Networks - GANs)

The Approach: This is a game of cat and mouse. You have an Artist trying to paint a fake Bach, and an Art Critic trying to spot the fake. The Artist tries to fool the Critic, and the Critic tries to get better at spotting fakes. They train together, getting better and better.
The Result: This was the toughest class. The training was unstable, like trying to balance a broom on your finger while someone keeps bumping the table. While the "Artist" got good at capturing local details (like the pitch of individual notes), the final music sounded more like modern jazz improvisation than structured Bach. It captured the sound but missed the style.

The Final Verdict

The researchers concluded that:

The Step-by-Step student (LSTM with Attention) was the clear winner. It was the most reliable and produced the most beautiful, stylistically accurate Bach-style music.
The Compressed Summary students (VAEs) struggled to keep the "secret code" useful, though the "block-based" version (VQ-VAE) showed promise.
The Art Critic game (GANs) was too difficult to train reliably for this specific style and ended up sounding more like jazz than Bach.

In short, when it comes to teaching a computer to write Bach, the most straightforward method—learning the notes one by one while paying attention to the big picture—worked best. The more complex methods of compressing the music or playing a game of "fake it till you make it" were interesting but less successful at capturing the true spirit of the master.

Technical Summary: Generative Modeling of Bach-Style Symbolic Music

Problem Statement

This study investigates the efficacy of three distinct families of generative models in producing symbolic piano music in the style of Johann Sebastian Bach. The core challenge involves modeling polyphonic note sequences that adhere to the complex counterpoint and structural elegance characteristic of Baroque music. The authors aim to compare the ability of these models to learn useful latent representations, generate stylistically coherent compositions, and avoid common failure modes such as mode collapse or posterior collapse. The dataset consists of Bach's known piano works (e.g., Well-Tempered Clavier, Goldberg Variations) converted into MIDI format.

Methodology

Data Pre-processing

The authors processed MIDI files into sequences of multihot vectors of dimension 88 (representing piano keys), where a value of 1 indicates a note is played at a specific time step. To simplify the modeling task, velocity information was discarded. Time steps were dynamically determined based on the shortest note duration appearing in more than 20% of a piece (e.g., sixteenth or eighth notes), and long pauses were filtered out.

Model Families

The study evaluates three primary approaches:

Autoregressive Models:
- Baseline LSTM: A 2-layer LSTM (hidden size 512) generating sequences of length 32.
- LSTM with Attention: An enhanced autoregressive model utilizing 2 LSTM layers and an attention mechanism. This architecture employs multiple parallel output heads to predict different voices of the piece simultaneously. The attention mechanism allows the model to dynamically focus on relevant parts of the input sequence, addressing limitations in handling long sequences where specific notes are more influential.
Latent-Variable Models (VAEs):
- Recurrent VAE: A standard VAE using a bidirectional LSTM encoder and a unidirectional LSTM decoder.
- Hierarchical Recurrent VAE: Inspired by Roberts et al. [2019], this architecture divides the decoder into a "conductor" (top layer) and a "decoder" (bottom layer). The conductor generates embeddings for subsequences, which are then used by the lower-level decoder to generate notes, forcing a stronger dependence on the latent variable $z$ .
- Vector-Quantized VAE (VQ-VAE): To mitigate posterior collapse, the authors implemented a VQ-VAE (van den Oord et al. [2017]). Input sequences were grouped into 4-note patterns. The model learns a discrete codebook of embedding vectors. An autoregressive LSTM is then trained on top of this quantized latent space to predict the sequence of embeddings, which are subsequently decoded into notes.
Adversarial Models (GANs):
- Wasserstein GAN (WGAN): The generator and discriminator both utilize LSTM and Multi-Head Attention architectures. The generator takes a latent vector sampled from a normal distribution as input. The authors employed the Wasserstein loss with gradient clipping to stabilize training, addressing issues of vanishing gradients and mode collapse common in standard GANs.

Key Results

Autoregressive LSTM with Attention: This approach yielded the most musically coherent samples. The model successfully captured stylistic nuances, producing compositions with sensible chord progressions and melodic structures reminiscent of the Baroque style. The attention mechanism proved critical in prioritizing salient features within sequences.
Latent-Variable Models:
- Standard and Hierarchical VAEs suffered from posterior collapse, where the model ignored the latent variable $z$ . While the hierarchical decoder delayed this collapse slightly, the resulting music lacked the specific stylistic fidelity of Bach, sounding more like jazz.
- VQ-VAE demonstrated success in mitigating posterior collapse. By learning discrete 4-note patterns, it produced outputs with clearer structural patterns (scales, arpeggios) and a semblance of Baroque style, outperforming continuous VAEs.
Generative Adversarial Networks: The WGAN successfully captured local pitch patterns and generated two distinct soundtracks (bass and treble). However, the model struggled with style generalization, producing music that resembled modern jazz piano with complex harmonies rather than Bach's style. The authors noted that GANs were difficult to train and did not reconstruct training data as efficiently as autoregressive or VAE-based models.

Significance and Claims

The paper concludes that for symbolic music generation in this context, autoregressive formulations are the most effective, while adversarial and latent-variable approaches present significant training challenges.

Relative Strengths: The autoregressive approach (specifically LSTM with attention) is identified as the "easiest" formulation, yielding the best results in terms of stylistic coherence.
Failure Modes: The study highlights specific failure modes: posterior collapse in VAEs and training instability/style mismatch in GANs.
Architectural Insights: The authors claim that vector quantization is a viable strategy to overcome the limitations of continuous latent spaces in VAEs. However, they assert that significant architectural changes are likely required to make latent-variable and adversarial models as robust as autoregressive ones for this specific task.
Future Work: The paper modestly suggests future directions, including the incorporation of broader musical features (dynamics, articulation) and the development of human-friendly latent structures (scales, emotion), but does not claim these have been achieved in the current study.

The authors emphasize that their work serves as a comparative study of these model families, highlighting that while all models could generate music, the autoregressive approach currently offers the highest degree of stylistic fidelity to Bach's work.