🤖 machine learning

ITGPT: A Transformer Based Architecture for the Generation of Dance Dance Revolution and In the Groove Charts

The paper introduces ITGPT, a transformer-based architecture designed to automate the generation of Dance Dance Revolution and In the Groove charts, achieving significant improvements in both accuracy and computational efficiency compared to previous methods.

Original authors: Miguel O'Malley

Published 2026-07-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Miguel O'Malley

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a DJ trying to remix a song for a video game where players have to dance on a giant floor with four glowing arrows. The game, called Dance Dance Revolution (or its cousin In the Groove), doesn't just play music; it needs a "chart." This chart is like a musical roadmap, telling the player exactly when to step on the Up, Down, Left, or Right arrows to match the beat. Making these charts by hand is like trying to write a complex poem while juggling; it takes hours, requires a deep ear for rhythm, and is incredibly tedious. This is where computer science steps in, specifically a field called machine learning, where we teach computers to learn patterns from data. In recent years, a type of AI called a "Transformer" has become famous for writing stories and translating languages because it's great at understanding the big picture of a sentence. But can a computer learn to choreograph a dance? That's the big question this paper tackles: can we build an AI that listens to a song and instantly invents a fun, playable dance routine that feels like it was made by a human expert?

Enter ITGPT, a new AI architecture designed specifically to solve this dance-floor puzzle. The researchers, led by Miguel O'Malley, propose that the old way of teaching computers to do this—using a mix of simple pattern detectors and memory loops—is like trying to navigate a city with a map that only shows the street you are currently standing on. It works for a few steps, but you quickly get lost in the bigger picture. Instead, ITGPT uses a Transformer architecture, which is like giving the computer a bird's-eye view of the entire song at once. This allows the AI to understand not just the next beat, but how the whole song flows, how fast it is (measured in BPM, or beats per minute), and how hard the dance should be.

The paper introduces a two-part system to generate these charts. First, the Step Placement model acts like a conductor, listening to the music and deciding when the arrows should appear. It looks at the audio and asks, "Is this a heavy drum hit? Is this a quiet melody?" and places a step accordingly. The second part, the Step Selection model, acts like the choreographer, deciding which arrow (Up, Down, Left, or Right) the player should hit. It looks at the steps already placed and the surrounding music to pick the best move, ensuring the dance feels natural and not random.

The results are quite promising. The authors found that ITGPT is significantly better at creating accurate charts than previous methods. In fact, it improved the accuracy of step placement and selection compared to its predecessors, DDC and DDCL. One of the most exciting findings is speed. While older models took a long time to generate a chart because they had to process the music and the dance moves in a slow, step-by-step loop, ITGPT is incredibly fast. The paper notes that ITGPT can generate a chart in about 0.06 seconds, which is roughly 7 times faster than the previous best model (DDCL) and nearly 50 times faster than another transformer-based model called GOCT. This speed boost happens because ITGPT processes the entire audio file once at the beginning, rather than re-listening to it for every single step.

However, the paper is careful not to claim perfection. The researchers admit that while ITGPT is a major leap forward, it still struggles slightly with "held notes" (steps where you have to keep your foot on an arrow for a while), a task where an older model called DDCL actually performed slightly better. They suggest this might be because the older model was better at spotting long, sustained sounds in the music. The authors also point out that their training data, while expanded to include 253 songs and 952 charts, is still relatively small compared to the massive datasets used for language models. They suspect that feeding the AI even more dance data would make it even smarter, but for now, ITGPT stands as a powerful, fast, and highly accurate tool for automating dance game creation.

In short, this paper suggests that by switching to a Transformer-based approach and giving the AI a "global view" of the music, we can generate dance charts that are not only more accurate but also created in the blink of an eye. It's a step toward a future where anyone could upload a song and instantly get a custom dance game, turning the tedious work of charting into a seamless, automated experience.

Technical Summary: ITGPT

Problem Statement
The generation of charts (choreography) for rhythm games like Dance Dance Revolution (DDR) and In the Groove (ITG) is a complex, manual process requiring the alignment of steps with musical activity, difficulty levels, and tempo (BPM). Previous automated approaches, such as Dance Dance Convolution (DDC) and its successor DDCL, utilized hybrid CNN-LSTM or ConvLSTM architectures. While effective, these models struggled with global context integration and computational efficiency, particularly in step selection. The challenge lies in creating a system that can accurately place steps (onset detection) and select specific arrow combinations (step selection) while adhering to musical structure, difficulty constraints, and BPM, all with reduced computational cost.

Methodology
The paper introduces ITGPT, a transformer-based architecture that replaces the LSTM components of previous works with a transformer framework to better encode global context. The system operates via a two-stage pipeline:

Step Placement (Onset Detection):
- Architecture: A hierarchical transformer encoder coupled with specialized prediction heads and multi-scale local refiners. The model processes audio features (80-band mel-spectrograms sampled at 32 timesteps per beat) alongside auxiliary features (BPM and fine integer difficulty).
- Hierarchical Processing: The model analyzes the chart across multiple time horizons, processing individual beats, 4-beat bars, 4-bar phrases, and 8-bar phrases before projecting to a compressed representation for global self-attention.
- Diagnostic Network: A novel auxiliary network is trained alongside the main model to estimate BPM and difficulty from the generated chart outputs. This enforces alignment with the input parameters during training, reducing the need for post-generation threshold tuning.
- Output: The model outputs a 48-length binary vector per beat, representing potential step placements. It utilizes a one-shot structure, generating all placements for a song chunk simultaneously.
Step Selection:
- Architecture: An autoregressive transformer model that selects one of 256 possible step combinations (4 arrows $\times$ 4 states: no step, step, hold, release) for each placed step.
- Input: The model takes the full context of previously placed steps (up to 500 steps) and audio features sampled in a radius around each step.
- Encoding: Audio is processed via a convolutional frontend followed by Residual Vector Quantization (RVQ) to discretize and contextualize audio representations. The model employs pervasive FiLM conditioning to integrate audio and step history tensors.
- Training Strategy: The model uses a multi-head prediction approach (predicting the current step and the next three) inspired by ProphetNet, with a dynamic weighting scheme that shifts focus from early to later steps as training progresses.

Key Contributions

Transformer Architecture: A shift from CNN-LSTM/ConvLSTM to a hierarchical transformer for step placement, enabling the capture of global musical context and structural patterns.
Diagnostic Model: The introduction of a diagnostic network to enforce consistency between generated charts and their intended BPM/difficulty parameters, eliminating the need for extensive threshold tuning during generation.
Efficient Step Selection: A transformer-based step selection model that processes audio once per chart and iterates purely over tokens for subsequent steps, significantly reducing generation time compared to autoregressive audio-processing models.
Expanded Dataset: The utilization of an expanded Fraxtil dataset (253 songs, 952 charts), roughly three times the size of the original dataset used in prior works, to improve model generalization.

Results

Step Placement: ITGPT achieves a mean F1-score of 0.7801, outperforming DDCL (0.7033) and DDC (0.5006). Notably, ITGPT shows the most significant improvement in lower difficulty levels (Beginner to Easy). The diagnostic model variant (ITGPT) requires minimal threshold tuning compared to predecessors, whereas the non-diagnostic variant (ITGPT-ND) shows performance gains similar to DDCL when thresholds are optimized.
Step Selection: ITGPT achieves an average accuracy of 0.5908 and a Top-3 accuracy of 0.9033, surpassing DDCL (0.5533) and DDC (0.4523). While DDCL slightly outperforms ITGPT in predicting "held notes" (likely due to its ConvLSTM's ability to capture sustained tones), ITGPT excels in general step prediction.
Generation Speed: ITGPT demonstrates a dramatic reduction in computational cost. The step placement model generates a chart in 0.06 seconds on average. The full pipeline (including step selection) is approximately 7× faster than DDCL and nearly 50× faster than the autoregressive GOCT model, while maintaining higher accuracy.

Significance and Claims
The authors claim that ITGPT represents a major departure from previous rhythm game chart generation methods. By leveraging transformer architectures, the model successfully integrates global musical context and auxiliary parameters (BPM, difficulty) more effectively than local-context models. The paper asserts that the hierarchical design is bespoke for the temporal structure of beat-delineated music, and the diagnostic model provides a novel approach to ensuring parameter consistency in rhythm game generation. The system achieves state-of-the-art performance in both accuracy and generation speed, making automated chart generation more viable for a wider range of difficulty levels and musical complexities. The authors note that while larger datasets (such as those used for osu!) could further improve performance, the current results demonstrate the efficacy of the proposed architecture specifically for DDR/ITG charts.

Technical Summary: ITGPT

More like this