⚡ electrical engineering

Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners

The paper introduces NAPE, a minimalist self-supervised learning framework that achieves state-of-the-art performance on audio and speech tasks by training a causal Transformer to predict the next patch embedding of a log-mel spectrogram, demonstrating that a simple autoregressive paradigm without complex pre-training recipes can effectively learn scalable audio representations.

Original authors: Umberto Cappellazzo, Xubo Liu, Stavros Petridis, Maja Pantic

Published 2026-08-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Umberto Cappellazzo, Xubo Liu, Stavros Petridis, Maja Pantic

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Sound is a story told in time. Unlike a photograph, which captures a single moment frozen in space, an audio signal unfolds as a sequence of events, one after another, carrying meaning through its rhythm and progression. For decades, computers have struggled to understand this story, often relying on complex, multi-step recipes to learn what a voice or a siren sounds like. These methods usually involve teaching a machine to reconstruct a sound from a broken version of itself, or to compare a sound against a teacher's example. While these approaches have worked, they require elaborate setups and heavy computational tools to reach their full potential.

A different path has emerged from the worlds of language and vision, where the most powerful artificial intelligence systems now learn by predicting what comes next. Instead of trying to rebuild a picture or a sentence from scratch, these systems look at what they have seen so far and guess the next piece. This simple act of anticipation forces the model to understand the underlying rules of the data, whether it is the grammar of a language or the structure of a visual scene. Researchers have long wondered if this same straightforward logic could work for sound, given that audio is naturally sequential. A team of scientists at Imperial College London and the University of Surrey has now answered that question with a new method that strips away the complexity of previous audio systems to focus on this single, forward-looking prediction.

The researchers introduced a framework they call NAPE, which stands for Next-Audio-Patch-Embedding prediction. To understand how it works, imagine taking a visual representation of sound, known as a spectrogram, which looks like a map of frequencies changing over time. The system breaks this map into small, square tiles. It then feeds these tiles into a computer model one by one, in a specific order that respects the flow of time. The model's only job is to look at the tiles it has already seen and predict the mathematical representation of the very next tile. It does not try to rebuild the original sound wave, nor does it compare its guess to a human-labeled example. It simply tries to get the next step right.

This approach is deliberately minimalist. Most modern audio learning systems rely on a "teacher-student" setup, where a large, frozen model guides a smaller one, or they use complex decoders to reconstruct the raw audio from hidden features. NAPE discards all of these extra components. It uses a single model that acts as both the observer and the predictor. To ensure the model learns the structure of the sound rather than just memorizing the input, the system prevents the model from seeing the future. It also uses a specific mathematical trick that stops the model from simply copying the current tile to predict the next one, forcing it to actually understand the relationship between the past and the future. The only signal the model receives is a measure of how close its prediction was to the actual next tile, calculated by comparing the direction of the two mathematical vectors in a high-dimensional space.

The results of this simple design were surprisingly powerful. The researchers tested NAPE on six different benchmarks, ranging from identifying environmental sounds like rain or dogs barking to recognizing spoken commands and detecting emotions in speech. They trained the system on a massive dataset of unlabeled audio clips from the internet. When they tested the model on these tasks, it achieved state-of-the-art performance, matching or beating the most complex systems currently available. This success held true across three different sizes of the model, from a small version with roughly 19 million parameters to a large version with over 300 million. The larger the model became, the better it performed, showing that the method scales up effectively without hitting a wall of diminishing returns.

One of the most revealing aspects of the study was how the model learned to organize the information. Because the system was trained only to predict the next piece of sound, it developed an internal map of audio that made sense on its own. When the researchers looked at where the model focused its attention, they found it naturally grouped together sounds that shared similar acoustic qualities, even though it had never been told what those sounds were. It learned to track how a specific frequency evolves over time, connecting the past to the present in a way that mirrors how humans perceive sound. This suggests that the model had discovered the fundamental structure of audio through the simple act of anticipation, without needing any human labels or complex reconstruction tasks.

The study also explored how the order in which the sound tiles were presented affected the learning. Since sound has a strong time axis, the researchers tested different ways of scanning the spectrogram tiles. They found that scanning the tiles in a way that respected the flow of time, such as moving left-to-right and bottom-to-top, worked best. This confirmed that the temporal nature of audio is the key to making this predictive approach work. When they tried to scan the tiles in a way that ignored the time sequence, the model's performance dropped significantly, proving that the method relies on the natural progression of sound.

Perhaps most importantly, the researchers found that the features learned by NAPE were highly useful even when the model was not fully retrained for a specific task. In a test where they froze the model and only trained a simple classifier on top of it, the system still performed very well. This indicates that the model had learned a robust and flexible representation of sound that could be easily adapted to new problems. While the model was not designed to be a perfect classifier, the fact that it could be used so effectively with minimal additional training suggests that the predictive approach captures the essence of audio more directly than previous methods.

The work demonstrates that the complex, multi-stage recipes that have dominated audio machine learning for years may not be necessary. By returning to a simple, causal objective—predicting the next step in a sequence—the researchers built a system that is both easier to train and more effective at learning. The model does not need a teacher, a decoder, or a massive set of auxiliary rules to succeed. It simply needs to look forward and guess what comes next. This finding opens a new path for audio artificial intelligence, suggesting that the most powerful way to teach a machine about sound might be to let it listen to the future, one step at a time.

Technical Summary: Listening Forward (NAPE)

Problem Statement

Self-supervised learning (SSL) has driven significant progress in audio representation learning, yet existing methods increasingly rely on complex pre-training recipes. Current approaches typically depend on reconstruction decoders, separately trained acoustic tokenizers, student-teacher architectures with exponential moving average (EMA) updates, and auxiliary regularization losses. While these methods achieve competitive performance, they lack the simplicity and unified interface seen in recent advances in language modeling (autoregressive next-token prediction) and visual representation learning (joint-embedding predictive approaches).

A critical gap exists in audio SSL: despite audio's inherent temporal structure making it naturally suited for autoregressive prediction, no method has successfully applied the "next-embedding prediction" paradigm to audio. Existing audio SSL methods largely adapt paradigms from vision (e.g., masked spectrogram modeling), which do not fully leverage the sequential nature of audio signals.

Methodology: The NAPE Framework

The authors introduce NAPE (Next-Audio-Patch-Embedding prediction), a self-supervised framework that adapts the causal next-embedding prediction paradigm to audio. The core philosophy is to train a model not as a static feature extractor, but as a predictive system that models the data distribution by predicting the next element in a sequence.

Core Architecture

Input Representation: Raw waveforms are converted into log-mel spectrograms ( $1 \times F \times T$ ). These are split into non-overlapping square patches ( $P \times P$ ) and projected into a $d$ -dimensional embedding space via a patch embedding layer ( $f$ ), resulting in a 2D grid of embeddings.
Scanning Order: Unlike bidirectional models where linearization order is permutation-equivariant, NAPE's causal formulation requires a specific scanning order to define "past" vs. "future." The paper evaluates four strategies:
- Raster: Left-to-right, bottom-to-top (standard).
- Time-major: Bottom-to-top within each time column, then advance in time.
- Zigzag: Alternating row directions.
- Diagonal: Sweeping along anti-diagonals.
- Finding: Raster and Diagonal orders perform best, as they align with the temporal unfolding of acoustic events. Time-major underperforms.
Causal Transformer Encoder ( $h$ ): A Vision Transformer backbone processes the linearized sequence using causal self-attention (preventing attention to future patches). The architecture includes pre-norm design, Rotary Position Embedding (RoPE), LayerScale, and query-key normalization.
Predictor Head ( $g$ ): A lightweight head (specifically a SimSiam-style 3-layer MLP) decouples the representation space from the prediction space. It produces an estimate of the next patch embedding ( $\hat{z}_{t+1}$ ) based on the current and past embeddings ( $z_{\le t}$ ).
Training Objective: The model minimizes the negative cosine similarity between the predicted embedding $\hat{z}_{t+1}$ $\overset{z}{^}_{t + 1}$ and the actual target embedding $z_{t+1}$ $z_{t + 1}$ .
- Loss Function: $L = \frac{1}{N-1} \sum D(\text{stopgrad}(z_{t+1}), \hat{z}_{t+1})$ .
- Key Mechanisms: The framework relies on three pillars to prevent trivial solutions (collapse):
  1. Causality: The attention mask restricts prediction to past context.
  2. Prediction Shift: The model at step $t$ predicts $t+1$ , preventing identity mapping.
  3. Stop-Gradient: Applied to the target embedding to prevent the encoder from collapsing both sides of the loss to a constant.

Minimalist Design

NAPE intentionally avoids:

Reconstruction decoders.
Acoustic tokenizers (discrete targets).
Student-teacher setups (EMA teachers).
Auxiliary regularization losses.
The entire learning signal comes from the model's ability to anticipate the next embedding in the sequence.

Key Contributions

First Causal Next-Patch-Embedding Framework for Audio: NAPE is the first self-supervised audio framework built around causal next-patch-embedding prediction, offering a substantially simpler alternative to reconstruction-based and joint-embedding approaches.
Systematic Ablation of Design Axes: The authors identify the critical components for success in the audio setting:
- Scanning Order: Raster and Diagonal are superior to Time-major.
- Predictor Head: A lightweight SimSiam-style MLP outperforms heavier Transformer predictors.
- Target: Predicting patch embeddings (continuous) is effective; predicting raw mel content is comparable, but predicting encoder features causes divergence without extra regularization.
- Similarity: Negative cosine similarity is optimal; distance-based metrics (L1/L2) lead to collapse.
Scalability: The framework scales consistently across three encoder sizes (Small: ~19M, Base: ~85M, Large: ~303M parameters).
Structured Representations: Qualitative analysis shows NAPE learns structured attention patterns that reason about spectral context and temporal history, and groups acoustically similar patches without explicit labels.

Experimental Results

Experiments were conducted on six benchmarks: AudioSet-2M (AS-2M), AudioSet-20K (AS-20K), ESC-50, Speech Commands V1 (KS1), Speech Commands V2 (KS2), and IEMOCAP.

Fine-Tuning Performance: NAPE achieves state-of-the-art (SOTA) or competitive results across multiple tasks.
- NAPE-L (Raster) achieves 50.2 mAP on AS-2M, matching the strong baseline SSLAM (which uses a complex student-teacher setup with reconstruction decoders).
- NAPE-L (Diagonal) achieves 68.8% accuracy on IEMOCAP (speech emotion recognition), a +3.5 point improvement over the previous best (BEATsiter3).
- NAPE outperforms Audio-MAE and A-JEPA across most benchmarks, particularly at smaller scales (e.g., +2.6 mAP over Audio-MAE on AS-2M for the Small model).
Scaling Behavior: Performance improves consistently from Small to Large models. The gains are most pronounced on audio classification tasks, while keyword spotting tasks show smaller absolute gains due to saturation.
Linear Probing: NAPE yields strong linear-probing results, indicating that the learned representations remain discriminative even under strict feature-freeze evaluation. The best probing layers are found in the middle of the network (e.g., layer 6 of 12 for the Base model), suggesting top layers specialize for the prediction task while mid-layers retain general classification-relevant information.

Significance and Claims

The paper claims that NAPE demonstrates that a simple, causal next-embedding prediction paradigm is sufficient to train powerful audio learners. By removing the need for complex heuristics (EMA, reconstruction decoders, tokenizers), NAPE provides a unified and minimalist pre-training interface that transfers effectively across modalities.

The authors emphasize that audio's temporal structure makes it uniquely suited for this paradigm, more so than vision. The results suggest that autoregressive prediction of continuous embeddings can replace the elaborate "recipes" currently dominating audio SSL, offering a path toward simpler, more scalable, and effective self-supervised learning for audio. The work establishes that the model learns meaningful, structured features (evidenced by attention patterns and embedding clustering) solely through the objective of anticipating the next patch in a spectrogram.