💻 computer science

Think Proprioceptively: Embodied Visual Reasoning for VLA Manipulation

The paper introduces ThinkProprio, a method that converts proprioceptive state into text tokens for early fusion with task instructions in Vision-Language-Action models, enabling embodied state to guide visual reasoning and token selection while achieving performance comparable to or better than strong baselines with over 50% reduced inference latency.

Original authors: Fangyuan Wang, Peng Zhou, Jiaming Qi, Shipeng Lyu, David Navarro-Alarcon, Guodong Guo

Published 2026-02-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Fangyuan Wang, Peng Zhou, Jiaming Qi, Shipeng Lyu, David Navarro-Alarcon, Guodong Guo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to cook a complex meal. You give it a recipe (the instruction) and it looks at the kitchen counter (the vision). But here's the problem: most current robot brains ignore how their own arms are positioned, how heavy their grip is, or if their joints are tired. They only look at the picture and read the text, then guess what to do.

The paper "Think Proprioceptively" introduces a new way for robots to think, called ThinkProprio. It argues that for a robot to truly understand a task, it needs to "feel" its own body while it is looking at the world, not just after it has already decided what to do.

Here is how it works, using simple analogies:

1. The Problem: The "Late Arrival" Guest

In most robot systems, the robot's body data (called proprioception—like knowing your elbow is bent or your fingers are open) is treated like a guest who arrives at the party after the main conversation has started.

The Old Way: The robot looks at the image, reads the instruction, makes a plan, and then asks, "Oh, where are my hands right now?" This is too late. By the time it checks its body, it might have already picked the wrong object or moved too far.
The Result: The robot is slow and makes mistakes because it isn't "feeling" the task while it's thinking about it.

2. The Solution: "Text-ifying" the Body

ThinkProprio changes the game by turning the robot's body data into text tokens (like words in a sentence) right at the very beginning.

The Analogy: Imagine you are reading a story. Usually, you read the plot (the instruction) and look at the pictures (the vision). ThinkProprio adds a new sentence to the very first page of the book that says, "The hero's arm is currently raised, and their grip is tight."
Why this helps: Now, as the robot reads the story and looks at the pictures, it already knows its own physical state. It can use that knowledge to decide which parts of the picture are actually important.

3. The Magic Trick: The "Smart Spotlight"

The biggest innovation is how this system saves time. Usually, a robot looks at every single pixel in a camera image, which is like trying to read every word on a billboard while driving down the highway. It's exhausting and slow.

ThinkProprio uses the "body text" and the "instruction text" to act as a smart spotlight.

How it works: The robot asks itself, "Given that I am holding a tool and the instruction says 'push the button,' which parts of this image actually matter?"
The Result: It ignores 85% of the image (the background, the floor, the ceiling) and only keeps the 15% of the image that is relevant to the task (the button and the tool).
The Benefit: It's like switching from reading a whole library to reading just the one page you need. This makes the robot 58% faster and uses much less computer memory, without losing accuracy.

4. The "Voting" System

How does the robot decide what to keep? It uses a clever "voting" system.

The instruction and the body data vote on which parts of the image are important.
If the instruction says "pick up the red block" and the robot's hand is near the table, the "red block" gets a vote. The empty wall gets no votes.
The robot keeps the winners and throws away the rest.

5. What the Results Show

The authors tested this on two famous robot training grounds (CALVIN and LIBERO).

Better Performance: The robot got better at long, complex tasks (like stacking blocks or opening drawers) because it could "feel" its way through the steps.
Much Faster: Because it stopped looking at the whole picture and only looked at the important parts, it made decisions in 22 milliseconds (compared to 52 milliseconds for other top models).
Efficiency: It achieved these results while using significantly less computer memory (VRAM).

Summary

ThinkProprio is like giving a robot a sixth sense that it can use while it is reading and looking, rather than waiting until the end. By turning its own body position into "words" and using those words to filter out the noise in its vision, the robot becomes faster, smarter, and more efficient at doing physical tasks. It proves that to move well, a robot must think about how it moves from the very first moment of understanding.

Technical Summary: Think Proprioceptively: Embodied Visual Reasoning for VLA Manipulation

Problem Statement

Vision-Language-Action (VLA) models have emerged as a powerful paradigm for generalizable robot control by translating visual observations and natural-language instructions into executable actions. However, current VLA pipelines typically introduce proprioceptive signals (robot state, such as joint angles and end-effector pose) late in the processing stack, couple them weakly to perception, or ignore them entirely.

This late integration prevents the robot's physical embodiment from shaping the initial understanding of the task or influencing which visual tokens are attended to throughout the policy. Consequently, the model fails to leverage physically grounded state to determine which scene elements are relevant for spatially grounded actions. While recent works like $\pi^0$ and $\pi^{0.5}$ have begun to integrate proprioception earlier, the functional role of proprioception remains difficult to isolate due to coupled design choices regarding vision-language aggregation and action-head conditioning.

Methodology: ThinkProprio

The authors propose ThinkProprio, an architecture designed to "think proprioceptively" by treating proprioception as a first-class modality that actively participates in early visual reasoning. The core components are:

1. Proprioceptive State Encoding via Text Tokenization

Unlike methods that use learned Multi-Layer Perceptrons (MLPs) to project continuous state vectors into the VLM embedding space (which can introduce distribution mismatches), ThinkProprio discretizes continuous proprioceptive state values.

Discretization: Continuous state values are clipped and uniformly binned.
Tokenization: These bin indices are mapped to specific vocabulary token IDs from the pre-trained VLM's embedding table.
Integration: The resulting proprioceptive tokens are concatenated with the language instruction tokens at the input of the VLM. This allows the robot's state to participate in the same embedding space as language, enabling state-aware visual conditioning without additional projection layers.

2. Physically Grounded Token Selection

ThinkProprio introduces a mechanism to guide the selection of visual tokens based on both the task instruction and the robot's current proprioceptive state.

Query Generation: Instruction tokens ( $H_l$ ) and proprioceptive tokens ( $H_p$ ) are concatenated to form guidance tokens ( $H_q$ ).
Scoring: Visual tokens attend to these guidance tokens to form a "physically grounded query." A score matrix is computed by matching normalized queries to normalized visual tokens.
Selection Mechanism: The system employs a vote-based discrete selection with a straight-through estimator (STE). Gumbel noise is added to scores to allow for differentiable training. Tokens receiving votes are retained, while others are discarded.
Global Context: To prevent the loss of coarse scene information, a global context token (averaged over all visual tokens) is appended to the selected subset.

3. Architecture and Training

Backbone: The compact sequence of selected visual tokens, global context, proprioceptive tokens, and language tokens is processed by a pre-trained VLM backbone.
Action Head: The action head conditions on the fused features from the VLM via cross-attention.
Objective: The action head is trained using flow matching (a diffusion-based approach) to predict continuous action chunks. The training objective minimizes the error between the predicted velocity field and the target velocity.

Key Contributions

Systematic Ablation of VLA Design: The paper provides the first systematic comparison of design axes in VLA models, uncoupling vision-language aggregation, conditioning mechanisms, and proprioceptive encoding/entry points.
Proprioceptive Text Tokenization: The authors demonstrate that discretizing proprioception into text tokens and feeding them into the VLM input is more effective than using learned projectors or late-conditioning mechanisms. This leverages the VLM's pre-trained token interface.
Physically Grounded Token Selection: The method shows that joint instruction-and-proprioception guidance can effectively bias computation toward action-critical evidence. This allows the model to retain only ~15% of visual tokens while matching or exceeding the performance of models using full token sets.
Efficiency: By reducing the number of visual tokens processed by the VLM and action head, the approach significantly lowers computational costs.

Experimental Results

The method was evaluated on the CALVIN (ABC→D split) and LIBERO benchmarks, as well as real-world manipulation tasks.

Task Performance:
- CALVIN: ThinkProprio achieved an average completed chain length of 4.55, outperforming strong baselines like FLOWER (4.44) and $\pi^{0.5}$ (2.38). It showed particularly strong gains in long-horizon tasks (LH-5), reaching 82.1% success compared to 77.8% for FLOWER.
- LIBERO: ThinkProprio achieved an average success rate of 97.3%, competitive with the best baselines (LightVLA at 97.4%) and achieving the highest score on the LIBERO-Long suite (95.2%).
Computational Efficiency:
- Latency: ThinkProprio reduced end-to-end inference latency by 58% (from 52 ms to 22 ms on CALVIN) compared to the FLOWER baseline.
- Token Reduction: The model processes an average of only 15 visual tokens per timestep (down from 100 available), significantly reducing the sequence length for the VLM and action head.
- VRAM: Peak VRAM usage was reduced to ~~1.9 GB, a substantial decrease compared to OpenVLA (~~14.5 GB).

Significance and Claims

The paper claims that ThinkProprio successfully addresses the limitation of current VLA models where proprioception is treated as a late signal. By tokenizing proprioception and fusing it early with language instructions, the model can shape its visual reasoning process to prioritize embodiment-relevant evidence.

The authors assert that:

Proprioception should be explicitly provided and actively involved during task understanding, not just at the action generation stage.
Text tokenization of continuous state is a superior encoding strategy compared to learned projectors, as it aligns with pre-trained token embeddings and allows state to participate in the VLM's reasoning process.
Aggressive visual token reduction is feasible without performance degradation if the selection is guided by both instruction and robot state.
The approach yields strong performance on long-horizon tasks while significantly reducing inference latency and memory footprint, making it a viable path toward efficient, embodied robot control.

The paper concludes that while images provide many implicit state cues, proper integration of proprioception provides additional grounded information that is critical for complex manipulation, particularly in long-horizon scenarios requiring extended correction and contact dynamics.