🤖 AI

KineVLA: Towards Kinematics-Aware Vision-Language-Action Models with Bi-Level Action Decomposition

This paper introduces KineVLA, a novel vision-language-action framework that achieves precise and controllable robotic manipulation by decoupling invariant task goals from variable kinematic specifications through bi-level action decomposition and reasoning, validated by superior performance on both simulation and real-world benchmarks.

Original authors: Gaoge Han, Zhengqing Gao, Ziwen Li, Jiaxin Huang, Shaoli Huang, Fakhri Karray, Mingming Gong, Tongliang Liu

Published 2026-03-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Gaoge Han, Zhengqing Gao, Ziwen Li, Jiaxin Huang, Shaoli Huang, Fakhri Karray, Mingming Gong, Tongliang Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to set the dinner table.

The Old Way (Vanilla VLA):
You tell the robot, "Put the wine bottle on the shelf."
The robot understands the goal: the bottle needs to be on the shelf. But it doesn't care how it gets there. It might grab the bottle by the neck, spin it around wildly, and slam it down so the label faces the back wall. If you say, "Put the bottle on the shelf," again, it does the exact same thing. It's like a chef who only knows how to cook "soup" but never adjusts the seasoning, temperature, or presentation based on your specific request.

The Problem:
In the real world, humans are picky. We don't just want the bottle on the shelf; we want the label facing forward so we can read it, or maybe to the left to match the other bottles. We want the robot to grab it by the body, not the neck. Current robots are great at "what" to do, but terrible at "how" to do it precisely.

The New Solution: KineVLA
The researchers behind this paper built a new robot brain called KineVLA. Think of it as giving the robot a "two-brain" system and a "translator" to understand your specific, picky instructions.

Here is how it works, using simple analogies:

1. The Two-Brain System (Bi-Level Action)

Imagine you are directing a movie.

Brain A (The Director): This brain handles the big picture. It says, "Okay, the scene is: Put the bottle on the shelf." It ignores the tiny details and focuses on the main goal.
Brain B (The Stunt Coordinator): This brain handles the physics. It says, "Wait, the Director said 'label facing left.' So, the arm needs to rotate 90 degrees, grab the neck gently, and slide it in without bumping the glass."

Most old robots only had the "Director." KineVLA has both. It separates the Goal (what we want) from the Kinematics (the precise math of movement, like speed, angle, and direction). This allows the robot to achieve the same goal in many different, highly specific ways.

2. The Translator (Bi-Level Reasoning)

Before the robot moves its arm, it "thinks out loud" in a special two-step language.

Step 1 (The Summary): It translates your complex sentence into a simple goal.
- You say: "Grab the carrot by the bottom and put it on the left side of the plate."
- Robot thinks: "Goal: Move carrot to plate."
Step 2 (The Blueprint): It translates the specific details into a technical blueprint.
- Robot thinks: "Constraint: Grip bottom tip. Trajectory: Move left. Final Pose: Horizontal."

This "thinking out loud" is crucial. It forces the robot to pause and verify that its plan matches your specific request before it even starts moving. It's like a pilot reading a checklist before takeoff to ensure they aren't just "flying," but flying to the right destination.

3. The Training Data (The Practice Field)

To teach this robot, the authors didn't just show it generic videos. They created a massive library of "picky" scenarios.

They filmed robots doing the same task (like opening a drawer) but in 50 different ways: opening it 10%, 50%, or 100%; pulling it straight, or pulling it diagonally.
They labeled every single move with precise language: "Pull the handle slightly," vs. "Pull the handle all the way."

Why This Matters

Think of the difference between a remote control car and a Formula 1 car.

The remote control car (old robots) just goes forward or turns left. It's fine for a driveway.
The Formula 1 car (KineVLA) can adjust its suspension, tire pressure, and aerodynamics in real-time to handle a specific turn at a specific speed.

In summary:
KineVLA is a robot that finally understands the difference between "Put the cup on the table" and "Put the cup on the table, but make sure the handle is facing me." It achieves this by splitting its brain into a "Goal Planner" and a "Motion Specialist," and by forcing itself to "think" through the details before it acts. This makes robots much more useful for delicate, personalized tasks in our homes and workplaces.

1. Problem Statement

Current Vision-Language-Action (VLA) models, such as OpenVLA and $\pi_0$ , excel at goal-oriented robotic manipulation but suffer from a critical limitation: kinematic insensitivity.

The Gap: Existing models treat instructions describing the same high-level goal (e.g., "place the bottle on the rack") as interchangeable, generating relatively fixed motion trajectories. They fail to distinguish between fine-grained kinematic constraints embedded in natural language, such as specific orientations, grasp points (e.g., "grasp by the neck" vs. "grasp by the body"), or precise trajectory shapes (e.g., "move diagonally" vs. "move straight").
The Challenge: In real-world human-robot interaction, semantically equivalent goals often require distinct, non-interchangeable execution trajectories due to explicit kinematic constraints (position, orientation, distance, motion direction). Current models optimize for what to do (the goal) rather than how to do it (the precise motion), leading to rigid behaviors that cannot satisfy user-customized kinematic instructions.

2. Methodology: KineVLA

The authors propose KineVLA, a framework designed to explicitly decouple goal-level invariance from kinematics-level variability. The core components are:

A. Bi-Level Action Representation (Bi-Level RVQ-VAE)

Instead of using uniform action binning or single-level tokenization, KineVLA employs a Bi-Level Residual Vector Quantized Variational Autoencoder (RVQ-VAE) to decompose actions into two complementary latent spaces:

Goal-Level (Coarse-Grained): Captures low-frequency structural information and the semantic task goal (e.g., "grasp," "place").
Kinematics-Level (Fine-Grained): Captures high-frequency corrections, contact-rich transients, and precise motion parameters (e.g., direction, velocity, specific orientation).

Training Strategy: A two-stage training schedule is used. Stage I pretrains the coarse path on broad multi-task data to learn stable, low-frequency tokens. Stage II freezes the coarse path and trains a new residual path specifically on kinematics-rich data to specialize the fine-grained codebook for subtle motion variations.

B. Bi-Level Generation with Reasoning Tokens

To align language instructions with the hierarchical action representation, KineVLA introduces Bi-Level Reasoning Tokens as explicit, supervised intermediate variables.

Mechanism: The model generates a "Chain-of-Thought" (CoT) consisting of:
1. Coarse Reasoning: A natural language summary of the general task goal.
2. Fine Reasoning: A detailed breakdown of kinematic constraints (e.g., "grasp the neck," "orient label left").
Alignment: The model jointly generates these reasoning tokens and the corresponding discrete action tokens. To ensure the reasoning faithfully reflects the motor intention, the authors maximize the Conditional Mutual Information $I(\text{Reasoning}; \text{Action} | \text{Context})$ using an InfoNCE objective. This regularizes the model to prevent semantic disconnection between the text plan and the executed motion.

C. Architecture

Base Model: Built upon OpenVLA (Prismatic-7B backbone).
Input: Replaces standard coarse instructions with kinematics-rich natural language.
Output: A unified vocabulary emits both reasoning text and action tokens (offset to the tail of the text vocabulary), allowing for a unified loss function.

3. Key Contributions

New Task Formulation: Introduced a "Kinematics-Rich" VLA task where task goals remain invariant, but execution trajectories must adapt to fine-grained kinematic specifications encoded in language.
KineVLA Framework: Developed a novel model featuring:
- Bi-Level Action Representation: Decouples goals from kinematics via a two-stage RVQ-VAE.
- Bi-Level Reasoning: Uses explicit reasoning tokens to ground kinematic constraints, improving interpretability and control.
- Mutual Information Regularization: Ensures consistency between the generated reasoning and the executed actions.
Kinematics-Aware Datasets: Constructed three new datasets spanning simulation (LIBERO) and real-world robotics (Realman-75), containing 378K frames with:
- Kinematics-rich language instructions.
- Bi-level annotations (Coarse Goal + Fine Kinematics).
- Diverse scenarios involving object parts, action constraints, and target relations.

4. Experimental Results

Experiments were conducted on LIBERO (simulation) and Realman-75 (real robot) benchmarks, comparing KineVLA against strong baselines like OpenVLA, $\pi_0.5$ , and VQ-VLA.

Performance Metrics:
- Goal Success Rate: All models performed comparably, indicating that existing VLAs are good at achieving high-level goals.
- Kinematics Success Rate: KineVLA significantly outperformed all baselines. While baselines dropped in accuracy when fine-grained constraints were applied, KineVLA maintained high success rates (e.g., ~76.5% on LIBERO-Goal-Relabeled vs. ~52-65% for baselines in ablation/intervention tests).
Ablation Studies:
- Adding Bi-Level Representation (Bi-Rep) improved success rates by ~3-5%.
- Adding Bi-Level Reasoning (Bi-Rea) further improved performance, proving the necessity of explicit reasoning for grounding kinematics.
- Mutual Information (MI) optimization yielded the best results, confirming that aligning reasoning with action reduces execution drift.
Intervention Analysis: Replacing predicted reasoning tokens with random/mismatched tokens caused a drastic drop in kinematics-following success (e.g., from 76.5% to 52.4%), proving that reasoning tokens play a causal role in generating correct kinematic actions.
Efficiency: KineVLA's inference time (~96ms) is comparable to single-level VQ models and diffusion-based methods, showing that the bi-level design does not incur significant overhead.

5. Significance

Bridging the Gap: KineVLA addresses the critical gap between high-level semantic planning and low-level physical execution, enabling robots to perform customized, precise manipulations based on natural language.
Interpretability: The use of explicit reasoning tokens makes the model's decision-making process transparent, allowing users to verify that the robot understood specific kinematic constraints (e.g., "face left").
Scalability: The bi-level decomposition approach provides a scalable framework for handling complex, physics-aware tasks without relying on computationally expensive diffusion models or force sensors, making it suitable for a wide range of robotic applications requiring fine-grained control.
Resource Availability: The release of the Kinematics-Rich Datasets and code provides a crucial benchmark for the community to advance research in kinematically aware robotic manipulation.