💻 computer science

Ordered Action Tokens for Visuomotor Policy Learning

This paper introduces Ordered Action Tokenization (OAT), a learned method that discretizes continuous robot action chunks into structured, ordered sequences to achieve high compression and total decodability, thereby enabling flexible inference-time tradeoffs between cost and fidelity while consistently improving visuomotor policy performance across diverse simulation and real-world tasks.

Original authors: Chaoqi Liu, Yue Zhao, Haonan Chen, Xiaoshen Han, Jiawei Gao, Ehsan Adeli, Yilun Du

Published 2026-07-27

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Chaoqi Liu, Yue Zhao, Haonan Chen, Xiaoshen Han, Jiawei Gao, Ehsan Adeli, Yilun Du

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to cook a complex meal. You can't just tell it "make pasta" and expect it to work; you have to break the task down into tiny, specific steps: "grab the pot," "fill with water," "turn on the stove," "wait for boiling." In the world of robotics, this is called visuomotor policy learning. It's the process of teaching a robot to look at the world (vision) and move its arms to do things (motor control) by learning from examples.

To teach a robot, we usually use a special kind of AI called a Vision-Language-Action (VLA) model. Think of this model as a super-smart robot brain that has read every book and watched every video. But here's the tricky part: the real world is continuous and smooth, like a flowing river. Robots, however, are digital and think in discrete steps, like a series of still photos. To teach the robot, we have to chop that smooth river of movement into little chunks and turn them into a language the robot brain can understand. We call these little chunks of movement "tokens." It's like turning a long sentence into a string of Lego bricks. If the bricks are too small, the sentence is too long to build. If they are too big or messy, the robot can't figure out how to put them together to make a valid move.

For a long time, scientists have been trying to find the perfect way to chop up these robot movements into tokens. Some methods chop them up too finely, creating a massive, unwieldy list of instructions. Others chop them up in a way that looks good on paper but falls apart when the robot tries to actually move. The big question has been: How do we chop up robot movements so they are short, easy to understand, and always result in a valid action?

This is where a new paper comes in with a clever solution called Ordered Action Tokenization (OAT). The researchers, a team from Harvard and Stanford, realized that the way we chop up the movements matters just as much as the chopping itself. They proposed a new way to turn robot actions into a sequence of tokens that follows a very specific, helpful order.

Imagine you are describing a painting to a friend over the phone. If you start by saying "it's a blue sky, then a green tree, then a red house," your friend gets a vague idea. But if you start with the big picture—"it's a landscape with a sky, trees, and a house"—and then add details like "the sky is bright blue" and "the house has a red roof," your friend builds a much better mental image, step by step. That is exactly what OAT does for robots.

The team discovered that existing methods were missing a key ingredient: order. Some methods just listed every tiny detail of a movement one by one, which was too long. Others compressed the movement into a secret code, but the code didn't tell the robot which part of the movement was the most important. The new OAT method forces the robot to learn the "big picture" of the movement first, and then fill in the fine details later.

Here is how it works in practice:

The First Token is the Big Idea: The very first token the robot predicts tells it the general shape of the action. For example, "reach for the cup."
Later Tokens Add Detail: The next few tokens refine that idea. "Reach for the cup... slightly to the left... with a gentle grip."
The Magic of "Anytime" Decoding: This is the coolest part. Because the tokens are ordered from "big idea" to "fine detail," the robot doesn't have to wait for the whole list to finish. If the robot is in a hurry, it can stop after just the first few tokens and still get a "good enough" action to grab the cup. If it has more time, it can wait for the full list to get a perfect, precise grab. It's like a text message that makes sense even if you only read the first half.

The researchers tested this idea in two main ways. First, they used it to teach simple robot brains to move in computer simulations and even on real robots in the lab. They found that using OAT made the robots much better at their tasks compared to older methods. The robots could learn faster and make fewer mistakes.

Second, they tested OAT in huge, powerful AI models (the kind that can talk and see). Even when the robot wasn't using the tokens to move directly, but just using them to help the AI "think" about the task, OAT still helped. It turned out that having a clear, ordered structure for the actions helped the AI understand the goal better, like having a clear outline for an essay before you start writing.

The paper also showed that you can't just mix and match these ideas. If you try to use a "block" method (where the robot guesses a whole chunk of tokens at once) with a tokenizer that wasn't designed for it, the robot gets confused and performs worse. The "order" of the tokens has to match the way the robot is guessing them.

In short, this paper suggests that the secret to teaching robots isn't just about compressing data or making it shorter. It's about structure. By organizing robot movements into a sequence that goes from "coarse" to "fine," we give the robot a roadmap that is flexible, efficient, and much easier to follow. Whether the robot is a small arm in a lab or a giant AI brain in the cloud, having a clear, ordered way to describe "how to move" seems to be the key to making them smarter and more reliable. The authors found that this approach consistently improved performance across many different tasks, from stacking cups to opening drawers, proving that sometimes, the way you tell a story is just as important as the story itself.

Technical Summary: Ordered Action Tokens for Visuomotor Policy Learning

Problem Statement

Action tokenization serves as the critical interface between continuous robot control and sequence modeling in modern visuomotor policies. However, existing approaches fail to simultaneously satisfy three essential properties required for effective policy learning:

High Compression: Token sequences must be compact to minimize training difficulty and inference latency.
Total Decodability: Arbitrary token sequences generated by a policy must reliably decode into valid, executable continuous actions.
Ordered Token Structure: The token sequence must organize control-relevant information such that early tokens capture coarse, high-impact motion, while later tokens refine residual details. This "coarse-to-fine" structure aligns with the inductive bias of next-token prediction in autoregressive models and provides meaningful supervision for vision-language models (VLMs).

Current methods satisfy subsets of these desiderata but not all simultaneously. Per-dimension binning is totally decodable but produces prohibitively long sequences. Frequency-domain tokenizers (e.g., FAST) offer ordering but lack total decodability due to variable-length expansion issues. Learned latent tokenizers (e.g., QueST, ACodec) provide compression but often lack an explicit ordered structure that places global control information early, limiting their effectiveness as policy targets.

Methodology: Ordered Action Tokenization (OAT)

The authors propose Ordered Action Tokenization (OAT), a learned tokenizer designed to satisfy all three desiderata. OAT discretizes continuous action chunks into compact, totally decodable, and ordered token sequences.

Architecture

OAT employs an encoder-decoder architecture with a discrete bottleneck:

Register Bottleneck: The encoder uses learnable "register" tokens (inspired by ViT registers) to cross-attend to all positions in the continuous action chunk. These registers aggregate temporal information into a fixed-length sequence of latent states.
Finite Scalar Quantization (FSQ): The latent register states are discretized into a fixed vocabulary using FSQ, ensuring that every vocabulary index maps to a valid continuous action chunk (Total Decodability).
Decoder: A cross-attention Transformer decoder reconstructs the continuous action chunk from the quantized register states.

Training: Ordered Prefix Reconstruction

To induce the desired coarse-to-fine ordering, OAT utilizes Nested Prefix Reconstruction during training:

Nested Dropout: The training objective samples a reconstruction budget $K$ from a set of budgets (e.g., $\{1, 2, 4, \dots, H_l\}$ ). The decoder is trained to reconstruct the full action chunk using only the first $K$ tokens, with the remaining suffix masked out.
Information Allocation: This forces early tokens to carry coarse, globally salient control information necessary for a plausible action, while later tokens refine residual details. This creates a "progressive" representation where any prefix decodes to an executable action.
Register Flow Constraints: The authors introduce specific self-attention masks for the registers to support different generation schedules. They define two variants:
- $OAT_{sing}$ (Token-wise): Uses standard causal attention, imposing a strict dependency order on every token.
- $OAT_{pow2}$ (Power-of-Two): Uses block-causal attention where registers within a power-of-two group cannot attend to each other, only to previous groups. This enables scalable block autoregressive decoding.

Policy Interfaces

OAT is validated in two prevailing policy paradigms:

Autoregressive (AR) Policies: The policy generates OAT tokens sequentially (or in blocks), which are then detokenized into actions. The ordered structure supports variable inference budgets, allowing the system to trade inference cost for action fidelity.
Token Co-Training (TC) Policies: The VLM is trained to predict OAT tokens (supervision), while a separate flow-matching expert generates continuous actions from the detached VLM context. Here, the first OAT token acts as a "plan-like" summary of the entire action chunk, providing a strong global objective for the VLM's prefill representation.

Scalable Generation

The paper introduces a Block-wise Autoregressive (BAR) framework that unifies token-wise generation, one-shot parallel decoding, and intermediate block schedules. Specifically, the $OAT_{pow2}$ variant combined with a power-of-two generation schedule reduces the sequential policy call depth from $O(H_l)$ to $O(\log H_l)$ , significantly improving inference efficiency without sacrificing performance.

Key Contributions

Formalization of Desiderata: The paper identifies and formalizes high compression, total decodability, and ordered token structure as the three core requirements for action tokenization as a policy interface.
OAT Algorithm: The proposal of a learned tokenizer that satisfies these requirements through register-based compression, FSQ, and nested prefix training.
Scalable Decoding Framework: The introduction of a BAR framework that unifies various generation schedules, demonstrating how power-of-two grouping can drastically reduce inference latency.
Comprehensive Validation: Extensive experiments across three policy backbones (lightweight Transformer, PaliGemma2, Qwen3VL) and over 60 tasks (simulated and real-world) covering both AR and TC paradigms.

Experimental Results

Rate-Distortion: OAT traces a smooth curve of reconstruction error vs. token budget, demonstrating that prefixes of varying lengths decode to progressively more accurate actions.
Lightweight Policies: In closed-loop control with a fixed Transformer backbone, OAT consistently outperforms baselines (Bin, FAST, QueST, ACodec). Performance scales with token budget, with $OAT_{sing}^{8}$ achieving the highest success rates across LIBERO, RoboMimic, MetaWorld, and real-world tasks.
Autoregressive VLMs: OAT improves success rates for large VLMs (PaliGemma2, Qwen3VL). The power-of-two variant ( $OAT_{pow2}$ ) matches the performance of token-wise generation while requiring significantly fewer policy calls (e.g., 5 calls vs. 16 for full generation).
Token Co-Training: Even when tokens are not decoded at inference, OAT supervision leads to superior VLM representations. OAT achieves the highest average success in TC settings, outperforming QueST and ACodec on several benchmarks, validating the "plan-like" supervision hypothesis.
Ablations: Removing the ordered prefix training (nested dropout) significantly degrades performance, confirming that the coarse-to-fine structure is critical. Furthermore, mismatching the register attention structure with the generation schedule (e.g., using $OAT_{sing}$ with power-of-two blocks) reduces performance, highlighting the need for alignment between tokenizer design and inference schedule.

Significance and Claims

The paper claims that OAT provides a robust and flexible interface for visuomotor policy learning. By satisfying the triad of compression, decodability, and ordering, OAT enables:

Flexible Inference: Policies can adaptively choose token budgets to balance latency and fidelity.
Improved Learning: The ordered structure aligns with the inductive biases of autoregressive models and provides meaningful global supervision for VLMs in co-training setups.
Scalability: The integration with BAR allows for efficient inference on large models without the linear depth penalty of standard token-wise generation.

The authors conclude that while the study is scoped to specific benchmarks and backbones, the design principles of OAT offer a generalizable solution for the action tokenization problem, paving the way for more efficient and capable embodied AI systems. They note that future work could explore adaptive computation, where policies dynamically decide the necessary token budget based on task complexity or uncertainty.