🤖 machine learning

NAC: Neural Action Codec for Vision-Language-Action Models

This paper introduces the Neural Action Codec (NAC), a neural audio codec-inspired architecture that treats robot action trajectories as multi-channel 1D signals to achieve high-fidelity compression and superior downstream performance in Vision-Language-Action models compared to existing tokenization methods.

Original authors: Ahad Jawaid, Yu Xiang

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ahad Jawaid, Yu Xiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot arm to perform a delicate task, like stacking blocks or pouring a drink. The robot needs to move smoothly and precisely, but the computer brain controlling it (a "Vision-Language-Action" model) is used to thinking in words and pictures, not in the smooth, continuous flow of physical movement.

To make them talk to each other, we have to translate the robot's smooth movements into a language the computer understands: a list of discrete steps or "tokens." Think of this like converting a smooth, flowing song into a series of musical notes.

The Problem: The Old Way Was Clunky

Previously, scientists tried to do this translation in two main ways:

The "Binning" Method: Imagine trying to describe a smooth curve by only using a ruler with very few marks. You'd have to chop the movement into tiny, jagged steps. This creates a massive list of instructions that is slow to process and often inaccurate.
The "Frequency" Method: Some tried to compress the movement like a zip file, but they used tools designed for human speech. While this helped, it wasn't perfect because robot movements and human voices are fundamentally different.

The Solution: Borrowing from Music (NAC)

The authors of this paper, Ahad Jawaid and Yu Xiang, had a clever idea: Why not use the technology that compresses music?

Modern AI models for audio (like the ones that generate music or compress Spotify files) are incredibly good at taking a complex sound wave and turning it into a compact list of codes, then reconstructing the sound perfectly later. They call this a "Neural Audio Codec."

The team realized that a robot's movement is actually just a sound wave in disguise.

Audio: A wave of air pressure changing over time (high frequency).
Robot Action: A wave of joint angles changing over time (lower frequency).

They built a new system called NAC (Neural Action Codec). They took the "music compression" engine and repurposed it to "compress" robot movements.

The Twist: Don't Listen to the Robot

Here is the most important discovery the paper makes: You cannot teach a robot to move by listening to it.

Audio codecs are trained to sound good to human ears. They use a special filter called a "Mel-spectrogram" that mimics how humans hear pitch. The authors found that if they used this human-hearing filter for robots, the robot would fail completely. It's like trying to tune a guitar by listening to a dog bark; the tools are mismatched.

Instead, they stripped away the "human ear" filters and replaced them with simple math that just cares about accuracy. They told the AI: "Don't worry about how it sounds; just make sure the robot's hand ends up in the exact right spot."

How It Works (The Analogy)

Think of the robot's movement as a long, complex movie.

The Encoder (The Summarizer): The NAC system looks at the movie and breaks it down into a few key scenes (coarse structure) and then fills in the tiny details (fine movements). It turns the whole movie into a short, efficient list of codes.
The Decoder (The Projector): When the robot needs to move, it takes that short list of codes and uses a special projector (called an ISTFT head) to turn it back into a smooth, high-definition movie of movement.
The "Discriminator" (The Critic): To make sure the movement is smooth and not jerky, the system uses a "critic" AI that checks the movement against the original. If the movement looks too choppy, the critic yells "No!" and forces the system to try again until it's smooth.

The Results

The team tested this new method against the old ones in computer simulations and in the real world.

In Simulations: The robot using NAC was significantly better at completing tasks (like stacking blocks) than robots using the old methods. It was more precise and made fewer mistakes.
In the Real World: When they tried it on actual physical robots, NAC still won, achieving a 50% success rate compared to much lower rates for other methods.

The Bottom Line

The paper claims that by treating robot movements like audio waves and using the best tools from the music industry (but removing the "human ear" filters), we can teach robots to move much more efficiently. It turns a messy, continuous physical problem into a neat, organized list of instructions that modern AI can easily learn and predict.

In short: They taught robots to move by teaching them to "sing" their movements, but they made sure the robot sang in its own language, not in a human's.

Technical Summary: Neural Action Codec (NAC) for Vision-Language-Action Models

Problem Statement

Vision-Language-Action (VLA) models aim to map visual observations and natural language instructions directly to low-level robot control signals. To leverage the generalization and reasoning capabilities of pre-trained Vision-Language Models (VLMs), robot control is often formulated as an autoregressive sequence modeling task. However, this approach requires action tokenization: converting continuous physical action trajectories into discrete symbols.

Existing tokenization methods face a trade-off between compression, latency, and downstream performance:

Uniform Binning: Early methods used per-dimension binning, resulting in prohibitively long token sequences for high-frequency control.
Hand-Designed Compression: Approaches like FAST utilize frequency-domain compression (e.g., Discrete Cosine Transform) to shorten sequences but may miss complex, non-linear dynamics due to fixed priors.
Latency and Fidelity: Designing an optimal tokenizer is challenging because it must capture statistical regularities for effective distribution modeling while maintaining low latency (tokens per action chunk) and fast decoding.

While the audio generation domain has successfully addressed similar challenges (high-fidelity compression, low latency, and complex temporal distributions) using Neural Audio Codecs, these architectures have not been directly adapted for robotic actions. A key gap exists in understanding how to transfer audio codec principles—specifically Residual Vector Quantization (RVQ) and adversarial training—to the distinct domain of multi-channel, low-frequency kinematic signals.

Methodology: Neural Action Codec (NAC)

The authors propose NAC, a learned action tokenizer that adapts state-of-the-art neural audio codec architectures (specifically multi-scale RVQGANs) for robotic action tokenization.

1. Signal Representation and Architecture

NAC treats continuous action chunks as multi-channel 1D signals.

Input: A continuous action chunk $a \in \mathbb{R}^{B \times H_a \times D_a}$ is flattened into a 1D pseudo-waveform $w \in \mathbb{R}^{B \times 1 \times L}$ , where $L = H_a \cdot D_a$ .
Encoder: A fully convolutional SEANet-style encoder processes the pseudo-waveform. It uses stacked 1D convolutions with strided downsampling and residual blocks to produce a continuous latent representation $z$ .
Quantization: The latent $z$ is discretized using Multi-Scale Residual Vector Quantization (MRVQ). Unlike standard RVQ, NAC applies per-stage temporal pooling before quantization. This forces earlier stages to capture coarse, global trajectory structures over wider temporal windows, while later stages capture fine-grained, high-frequency residuals.
Decoder: A Vocos-style decoder with an Inverse Short-Time Fourier Transform (ISTFT) head reconstructs the continuous action signal from the discrete codes. The ISTFT head predicts STFT magnitude and phase, synchronizing with the encoder's hop length to recover phase information lost during encoding.

2. Critical Adaptations for Robotics

The paper identifies that direct application of audio codecs fails without specific modifications:

Loss Function Shift: Audio models typically optimize for mel-spectrogram losses to align with human auditory perception. The authors demonstrate that mel-spectrogram objectives are ill-suited for kinematic signals, causing downstream policy performance to collapse to near zero. NAC replaces these with simple time-domain (MSE/L1) and non-mel spectral reconstruction losses.
Adversarial Training: NAC introduces the first application of adversarial loss to action tokenization. A discriminator (specifically a DAC-style discriminator) is used to preserve high-frequency fidelity, which is critical for modeling rapid corrective motions and sharp velocity changes in robotics.
Commitment Loss: To prevent codebook collapse, a commitment loss is applied with aggressive scaling (e.g., factor of 1000) to strictly bound the latent space.

3. Policy Formulation

NAC enables an autoregressive behavioral cloning policy:

Offset Codebooks: To provide a structured token space, NAC uses offset codebooks. If there are $n_q$ quantization scales, the vocabulary size is $|V| = n_q \times V_{bins} + 1$ . Tokens for each scale are bounded within specific ID ranges.
Training: The policy predicts the discrete token sequence $C = [c_1, \dots, c_L]$ conditioned on visual observations and language instructions, maximizing the log-likelihood of the token sequence.
Inference: The policy generates tokens autoregressively. The tokens are partitioned by scale, converted back to code indices, and passed to the frozen NAC detokenizer to recover executable continuous actions.

Key Contributions

NAC Architecture: Introduction of a learned action tokenizer based on a multi-scale RVQGAN architecture, specifically adapted for multi-channel robot trajectories by treating actions as 1D pseudo-waveforms.
Structured Autoregressive Policy: Formulation of a behavioral cloning policy that utilizes NAC's offset codebooks for structured, causal next-token prediction.
Loss Function Insight: Demonstration that dropping audio-specific mel-spectrogram losses is critical for applying neural audio codecs to robotics. The paper introduces the use of time-domain and non-mel spectral losses, along with the first use of adversarial loss in action tokenization.
Empirical Validation: Comprehensive evaluation across simulated (LIBERO-10, RoboMimic) and real-world manipulation tasks, showing superior reconstruction fidelity and downstream policy performance compared to existing baselines.

Experimental Results

The authors evaluated NAC against continuous-control baselines (Diffusion Policy) and discrete-token baselines (Uniform Binning, FAST, VQ-VLA, OAT).

Reconstruction and Ablation:
- Mel-Spectrogram Loss: Using mel-spectrogram losses resulted in 0% success rate on downstream tasks, confirming the perceptual mismatch between audio and kinematics.
- Adversarial Training: Removing the discriminator caused complete downstream failure, highlighting the necessity of adversarial training for preserving fine-grained temporal details.
- Decoder Head: Replacing the ISTFT head with a linear decoder significantly worsened both reconstruction error and policy success.
Downstream Performance (Simulation):
- On LIBERO-10, NAC achieved a 49.73% success rate, outperforming FAST (38.02%) and OAT (44.17%).
- On RoboMimic, NAC achieved 33.94%, surpassing FAST (28.38%) and OAT (31.94%).
Real-World Performance:
- On a suite of 8 physical manipulation tasks, NAC achieved a 50.0% success rate, outperforming all baselines (e.g., FAST at 40.0%, OAT at 40.0%).

Significance and Claims

The paper claims that repurposed neural audio codecs offer a strong, practical backbone for learned action tokenization in modern VLAs. The significance lies in demonstrating that:

The core architecture of audio foundation models (convolutional encoders, RVQ, adversarial training) is highly effective for robotics, provided the frequency-domain objectives are adapted to kinematic signals.
Adversarial training is essential for capturing the high-frequency dynamics required for precise robot control.
NAC provides a compact, ordered token space that enables standard autoregressive policies to operate over short, structured sequences, achieving lower reconstruction error and higher success rates than binning, FAST, and prior VQ-based tokenizers at comparable or better compression rates.

The authors conclude that NAC bridges the gap between audio foundation models and robotic control, offering a data-driven alternative to hand-designed compression that preserves the structural fidelity necessary for effective policy learning.