Lost in Reconstruction: Aligning Action Representations with Language in Vision-Language-Action Models

Imagine you are teaching a robot to cook. You might think the most important thing to tell it is what the final dish should look like: "The soup should be hot and in the bowl." But what if the robot only cares about the final picture? It might try to get the soup into the bowl by throwing it through the air, splashing it everywhere, or even dropping the pot. To cook properly, the robot needs to understand not just the destination, but the dance of the movement itself—the gentle stir, the careful pour, the specific way a hand grips a handle. This is the world of Vision-Language-Action (VLA) models. These are super-smart AI brains that combine what a robot sees (vision), what it is told to do (language), and how it moves its body (action). For a long time, scientists have been great at teaching robots to understand the "what" and the "where," but they've struggled to teach the robot the "how." They've been trying to translate the robot's complex, continuous movements into a simple list of instructions, but in doing so, they might have accidentally deleted the most important part of the recipe: the style and nuance of the motion.

This paper, titled "Lost in Reconstruction," tackles a specific problem in how we teach these robot brains to move. The researchers found that when we turn a robot's smooth, flowing movements into a list of digital "tokens" (like turning a song into a list of musical notes), the standard method throws away the clues that tell the robot which verb is being performed. They discovered that words like "flip," "push," or "fold" carry hidden information in the way the robot moves, not just in where it ends up. By fixing the way these movements are translated into digital code, they created a new system called SALT (Semantically ALigned action Tokenizer). In their tests, robots using SALT were much better at following instructions, succeeding in tasks about 72% of the time compared to less than 43% for the old methods. The paper suggests that to make robots truly understand human language, we need to make sure their "movement vocabulary" keeps the meaning of the verbs alive, rather than just focusing on how accurately they can redraw the path.

The Problem: When "How" Gets Lost in "What"

Think of a robot's movement as a continuous, flowing river. To teach a robot using a Vision-Language-Action model, scientists often have to turn this flowing river into a series of stepping stones. They chop the movement into tiny chunks and assign each chunk a digital label, or "token." It's like turning a smooth jazz melody into a list of numbers so a computer can read it.

The problem is that the standard way of doing this is obsessed with reconstruction. It asks: "Can we turn these numbers back into the exact same river?" If the numbers are close enough to the original movement, the system says, "Good job!" But the researchers realized that being "close enough" isn't the same as being "meaningful."

Imagine you are describing a dance. You could say, "The dancer moved from point A to point B." That's the goal. But the verb—whether they "glided," "stomped," or "twirled"—is hidden in the details of the movement. The paper argues that current robot systems are like a translator who only cares about the destination. They might turn a "flip" and a "push" into the same set of stepping stones if both end up in the same spot, even though the movements are totally different. The system loses the "verb-grounding" information—the clues that tell the robot how to do the action.

The Discovery: Verbs Live in the Motion

To prove this, the researchers looked at a massive dataset of real robots being controlled by humans, where every movement was paired with a spoken instruction like "put the spoon on the towel" or "flip the pancake." They broke down the instructions into verbs and analyzed the robot's movements.

They found two distinct sources of information:

1. The Goal: The visual change between the start and the end (e.g., the spoon is now on the towel).
2. The Motion Dynamics: The actual path the robot took to get there (e.g., the speed, the curve, the timing of the gripper).

The paper shows that these two sources are complementary. For some verbs, like "fold," the final shape is the most important clue. But for others, like "flip" or "turn," the way the robot moves is the only thing that tells you what the verb is. You can't tell a "flip" from a "push" just by looking at the start and end points; you have to watch the motion. The researchers measured this and found that the motion dynamics provided unique information that the visual goal alone could not capture. In fact, for certain verbs, the motion carried about 0.059 bits of unique information—enough to make a huge difference in understanding the instruction.

The Mistake: Compression Erases Meaning

The researchers then tested how well different methods of turning movements into digital tokens preserved this "verb" information. They looked at three common ways of doing this:

• Bin: Simply chopping the movement into fixed-size buckets.
• FAST: Compressing the movement into frequency patterns.
• VQ-VAE: Learning a set of "stepping stones" by trying to rebuild the movement as accurately as possible.

They found that as these systems compressed the data to make it smaller (fewer tokens), they systematically erased the verb information. It was like squeezing a sponge: as you squeeze out the water (data), you also squeeze out the shape (meaning). The "reconstruction-only" methods, which only cared about rebuilding the path accurately, failed to keep the linguistic clues. The gap between the original movement and the tokenized version grew wider as the compression got stronger. The paper argues that this is a bottleneck: if the robot's internal language for movement doesn't match the human language for movement, the robot will struggle to follow instructions.

The Solution: SALT

To fix this, the authors introduced SALT (Semantically ALigned action Tokenizer). Instead of just teaching the robot to rebuild the movement, they added a new rule: "You must also be able to guess the instruction from the movement."

Here's how it works in a simple analogy: Imagine you are teaching a student to draw a picture.

• Old Method: You tell the student, "Draw this picture so it looks exactly like the original." They focus on getting the lines right.
• SALT Method: You tell the student, "Draw this picture so it looks like the original, and if I show your drawing to a friend, they should be able to guess what the picture is supposed to be."

In technical terms, SALT takes the digital "stepping stones" of the movement and feeds them into a frozen, pre-trained language model. The model has to guess the original instruction (like "flip the cup") based only on those stepping stones. If the stepping stones don't contain enough information to guess the instruction, the system gets a penalty. This forces the robot to organize its movement tokens in a way that keeps the "verb" information safe, even while it still learns to rebuild the movement accurately.

The Results: Better Robots, Clearer Meaning

The researchers tested this new system in a simulation called SimplerEnv, where a robot had to perform four different tasks, like putting a spoon on a towel or stacking blocks. They compared three versions of the robot:

1. One using the standard FAST tokenizer.
2. One using the standard VQ-VAE tokenizer (reconstruction only).
3. One using the new SALT tokenizer.

The results were clear. The robot with SALT succeeded in 71.9% of the attempts. The standard VQ-VAE robot only succeeded 42.7% of the time, and the FAST robot managed just 31.2%.

But it wasn't just about winning more games. The researchers looked inside the robot's "brain" to see how it organized its movement vocabulary. They found that SALT developed verb-specialized codes. For example, specific digital tokens became highly dedicated to the action of "flipping" or "turning." In contrast, the old methods scattered these movements across generic, mixed-up tokens. SALT even learned to group different ways of saying the same thing (like "turn the lever" and "move the lever vertical to the front") under the same movement code, showing it understood the meaning rather than just the surface words.

What This Means

The paper suggests that for robots to truly understand and follow human language, we can't just focus on making their movements look right. We have to make sure their internal representation of movement preserves the linguistic distinctions humans make. By aligning the robot's action tokens with the language that describes them, we can build robots that are not only more accurate but also more intuitive. The authors note that while these results are promising, they were tested in simulation with a specific dataset, and future work will need to see if these gains hold up in the messy, unpredictable real world. However, the core idea—that preserving the "how" of action is just as important as the "what"—offers a new path forward for making robots that truly understand us.