💬 NLP

A novel LSTM music generator based on the fractional time-frequency feature extraction

This paper proposes a novel AI-based music generation system that combines fractional Fourier transform (FrFT) for time-frequency feature extraction with a long short-term memory (LSTM) network trained on the GiantMIDI-Piano dataset to produce high-quality music comparable to human compositions.

Original authors: Li Ya, Chen Wei, Li Xiulai, Yu Lei, Deng Xinyi, Chen Chaofan

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Li Ya, Chen Wei, Li Xiulai, Yu Lei, Deng Xinyi, Chen Chaofan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot to compose beautiful piano music. You can't just tell it, "Play something nice." You have to teach it how music feels and moves over time. This paper describes a new, clever way to do exactly that using two main tools: a special mathematical lens called FrFT and a smart memory system called LSTM.

Here is the breakdown of how their "AI Music Chef" works, using simple analogies:

1. The Problem: Music is a Moving Target

Music isn't just a list of notes; it's a wave that changes constantly. If you look at a song only as a list of notes (like a recipe), you miss the flavor. If you look at it only as a sound wave (like a photo of a sound), you miss the ingredients.

The authors realized that to teach a computer to write music, they needed to see the song in a way that captures both the timing (when notes happen) and the frequency (what notes sound like) simultaneously, but with more flexibility than standard tools allow.

2. Tool #1: The "Magic Prism" (Fractional Fourier Transform - FrFT)

Think of a standard music analysis tool (like a regular Fourier Transform) as a prism that splits white light into a rainbow. It shows you all the colors (frequencies) present in the song, but it flattens them out. It tells you what colors are there, but not exactly when they appear in the flow of time.

The Fractional Fourier Transform (FrFT) is like a rotatable, magic prism.

Instead of just looking at the song from the front (time) or the side (frequency), this prism can tilt to any angle in between.
By tilting the prism to a specific "fractional" angle, the AI can see hidden patterns and local structures in the music that are invisible to standard tools.
The Result: The AI gets a much richer, more detailed "map" of the music. It sees not just the notes, but the texture and flow of the sound in a way that is easier for a computer to understand.

3. Tool #2: The "Super-Student" (Long Short-Term Memory - LSTM)

Once the "Magic Prism" breaks the music down into these detailed maps (splitting them into "Real" and "Imaginary" parts, which are just mathematical components of the signal), the AI needs someone to learn from them.

Enter the LSTM.

Imagine a student trying to learn a long, complex story. A normal student might remember the first sentence but forget it by the time they get to the tenth paragraph.
The LSTM is a super-memory student. It has special "gates" (like a bouncer at a club) that decide what to remember, what to forget, and what to pay attention to.
It reads the detailed maps created by the FrFT. It learns the rules of the song: "After this sad chord, a happy melody usually follows," or "This rhythm tends to speed up."
Because it has such a good memory, it can predict the next note in the sequence with high accuracy, even if the song is long and complex.

4. The Cooking Process: How They Put It Together

The authors built a system that works like a three-step kitchen:

Preparation (Input): They take a real piano song (from a dataset called GiantMIDI-Piano).
Chopping & Mixing (FrFT): They run the song through the "Magic Prism." This turns the audio into a complex mathematical signal with "Real" and "Imaginary" parts. Think of this as chopping ingredients into very specific, precise shapes that are easy to cook with.
Cooking (LSTM): The "Super-Student" LSTM eats these chopped ingredients. It learns the patterns. Then, it starts cooking up new ingredients (predicting the next notes) based on what it learned.
Serving (Output): The new notes the AI cooked up are run through the "Magic Prism" in reverse (to turn the math back into sound) and saved as a new music file.

5. The Taste Test: Did it Work?

The authors tested their system by having it learn from thousands of piano pieces and then trying to write new ones.

The Result: The music the AI generated was incredibly close to the original human-made music.
The Proof: They measured the "error" (how different the AI music was from the target). Their new method had a very low error rate (0.0155), meaning the AI's prediction was almost perfect compared to older methods (which had an error of 0.0351).

The Big Picture

This paper is essentially saying: "If you want a robot to write great music, don't just show it the notes. Show it the music through a special, tilting lens that reveals its hidden structure, and then let a computer with a super-memory learn the patterns."

The result is an AI that can compose high-quality, human-like piano music, which could help musicians find inspiration, help students learn music theory, or even create custom soundtracks for video games in real-time.

1. Problem Statement

The paper addresses the limitations of current AI-driven music generation systems, specifically:

Lack of True Creativity: Many deep learning models tend to mimic training data without generating novel, creative content.
Handling Non-Stationary Signals: Traditional Fourier Transforms (FFT) struggle with non-stationary signals (like music) where frequency content changes over time. They provide a global frequency view but lack precise time-frequency localization.
Long-Term Dependencies: Standard Recurrent Neural Networks (RNNs) often fail to capture long-term dependencies in sequential data due to vanishing gradient problems, which is critical for maintaining musical structure and coherence over time.
Feature Representation: Existing methods may not fully exploit the intricate interplay between time and frequency domains required to model complex musical nuances.

2. Methodology

The authors propose a hybrid framework combining Fractional Fourier Transform (FrFT) for feature extraction and Long Short-Term Memory (LSTM) networks for sequence modeling and generation.

A. Feature Extraction: Fractional Fourier Transform (FrFT)

Instead of using standard time-domain waveforms or traditional STFT, the system utilizes FrFT to analyze music signals.

Concept: FrFT is a generalization of the standard Fourier Transform that allows for signal analysis in a "fractional domain" (a rotation of the time-frequency plane by an angle $\alpha$ ).
Process:
1. Input: MIDI or WAV signals are processed.
2. Transformation: The music signal $f(t)$ is transformed into the fractional domain using FrFT.
3. Decomposition: The resulting complex signal is split into its Real and Imaginary parts.
4. Normalization: These parts are normalized and fed separately into the neural network.
Advantage: FrFT provides a more flexible representation of non-stationary signals, capturing local structures and nuances that standard transforms might miss.

B. Sequence Modeling: LSTM Network

The extracted features are processed by a deep LSTM network designed to handle sequential data and long-term dependencies.

Architecture:
- Input: The normalized real and imaginary parts of the FrFT-transformed signal.
- Structure: A multi-layer LSTM (4 layers) with a hidden state size of 256.
- Mechanism: The network utilizes input, forget, and output gates to manage memory cells, allowing it to retain information over long sequences (crucial for musical phrasing and structure).
Training Strategy:
- Dataset: The GiantMIDI-Piano dataset (high-quality solo piano recordings).
- Preprocessing: Data is sampled at 5000 Hz, transformed via FrFT ( $\alpha = 0.05$ ), and segmented into groups of 200 sampling points.
- Loss Function: Mean Squared Error (MSE) is used to minimize the difference between predicted and actual signal values.
- Hyperparameters: Learning rate of 0.0003, batch size of 32, trained for 30 epochs.

C. Generation and Reconstruction

Prediction: The LSTM predicts the next steps in the sequence based on the hidden layer features and real-time inputs.
Reconstruction: The predicted real and imaginary parts are merged, inverse-normalized, and converted back to the time domain using the Inverse Fractional Fourier Transform (IFrFT) to generate the final audio waveform.

3. Key Contributions

Novel Feature Extraction: The integration of FrFT for music feature extraction is a significant innovation. It moves beyond standard time-frequency analysis to capture signal characteristics in a fractional domain, offering a richer representation of musical signals.
Hybrid Architecture: The paper successfully combines FrFT for spectral analysis with LSTM for temporal sequence prediction, addressing both the spectral complexity and temporal dependencies of music.
Ablation Study: The authors conducted a comparative analysis between their FrFT-LSTM method and a baseline LSTM trained directly on raw signals.
- Baseline Loss: 0.0351
- Proposed Method Loss: 0.0155
- This demonstrates that the fractional domain feature extraction significantly reduces error and improves prediction accuracy.
Real-Time Capability: The system is designed to generate music in real-time, allowing for interactive creation and improvisation.

4. Experimental Results

Convergence: Training curves (Figure 5) show successful convergence for both the real and imaginary part networks, indicating stable learning.
Signal Fidelity: Visual comparisons of the generated waveforms (via IFrFT) against original signals (Figure 8) show a high degree of resemblance. The red curves (generated) closely follow the blue curves (original).
Performance: The system successfully predicted both the real and imaginary components of the music signals, validating the model's ability to reconstruct complex audio structures.
Dataset: Experiments were validated using the GiantMIDI-Piano dataset, specifically testing on pieces like "Je t'aime Juliette."

5. Significance and Applications

Music Composition: The system can generate high-quality, human-comparable music, serving as a tool for inspiration for composers or for creating original works in various styles (pop, jazz, classical).
Music Education: It can be used to create tailored exercises, demonstrate music theory concepts, and help students understand structural patterns in music.
Interactive Entertainment: The real-time generation capability opens doors for interactive duet improvisation between humans and machines, as well as dynamic music generation for video games based on player preferences.
Technical Advancement: The study highlights the potential of fractional calculus in signal processing for AI applications, suggesting a new direction for handling non-stationary signals in deep learning.

In conclusion, the paper presents a robust framework where FrFT enhances the input representation of music signals, and LSTM effectively models the temporal evolution of these signals, resulting in a music generator that outperforms baseline methods in terms of signal reconstruction accuracy and structural coherence.