💻 computer science

ValueFormer: A Causal Transformer Value Function with Stage-Aware Labels for Semi-Autonomous Vision-Language-Action Policies

ValueFormer addresses the silent failure of imitation-learning-based Vision-Language-Action policies by introducing a compact, causal transformer that generates dense, stage-aware per-frame value labels to enable effective online mistake detection and advantage estimation, thereby significantly improving task completion rates on real-robot manipulation tasks without requiring costly reinforcement learning.

Original authors: Inkyu Sa, Konstantin Stulov, Rajat Bhageria

Published 2026-08-05

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Inkyu Sa, Konstantin Stulov, Rajat Bhageria

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where robots are learning to do chores not by being programmed with rigid rules, but by watching humans and copying them, much like a toddler learning to tie their shoes by watching a parent. This field is called Vision-Language-Action (VLA), where a robot uses its "eyes" (cameras) and "brain" (language models) to decide what to do next. The big problem with this approach is that the robot often learns only from successful examples. It's like trying to learn to ride a bike only by watching videos of people who never fell down. If the robot starts to wobble or drop the handlebars, it doesn't know it's failing until the whole thing crashes. It has no internal "gut feeling" to tell it, "Hey, you're doing great!" or "Stop! You're about to drop that sandwich!"

This paper tackles that missing "gut feeling." The researchers wanted to build a second, smaller brain for the robot that acts like a progress bar. Instead of just waiting for the robot to finish a task and then saying "Good job" or "Fail," this new system watches the robot in real-time, frame by frame. It tries to answer: "How close are we to the finish line right now?" and "Did we just make a mistake?" The goal is to make semi-autonomous robots safer and more helpful, so a human operator doesn't have to stare at a screen waiting for a disaster to happen.

The Robot's "Progress Bar" That Knows When You're Stuck

Meet ValueFormer, a new tool designed to give robots a sense of progress. Think of a robot trying to make a sandwich. It has to put down the bottom bun, add a patty, cheese, tomato, lettuce, and finally the top bun. If the robot gets stuck on the lettuce, a standard robot trained only on success stories might just keep trying to grab the lettuce forever, or worse, skip it and put the top bun on a bare patty, thinking everything is fine. It has no idea it's in trouble.

The authors of this paper realized that the biggest hurdle wasn't building a fancy new robot brain; it was figuring out how to teach the robot what "doing well" looks like at every single second.

The Problem with "All or Nothing" Labels

Imagine you are grading a student's math test. If you only give them a grade at the very end, you don't know when they got stuck. Did they fail the first question, or the last one?

The Old Way: Most robots are trained with a simple "Success/Fail" switch at the end of the task. If the sandwich is made, it's a 1. If it's a mess, it's a 0.
The Flaw: This is too sparse. If a robot makes a mistake halfway through but keeps going, the "0" at the end doesn't tell the robot which part was bad. It's like telling a runner, "You lost the race," without telling them they tripped at mile 2.

The researchers tried a different approach. They wanted a dense label, a score that updates every fraction of a second, showing exactly how close the robot is to the goal. But here's the tricky part: if you just tell the robot "You failed, so your score is 0" for the entire failed attempt, the robot gets confused. It sees the first few seconds (where it was doing fine) and thinks, "Wait, I was doing great, but you're saying I'm failing? I must be failing from the start!" This confuses the robot and makes it perform poorly even on good attempts.

The "Success-Then-Decay" Solution

The paper's main breakthrough is a clever way of labeling mistakes called "Stage-Aware, Success-Then-Decay."

Imagine the robot is climbing a mountain (making a sandwich).

The Climb: As it climbs, its "progress score" goes up smoothly.
The Slip: If it slips and falls (makes a mistake), the score doesn't instantly drop to zero. Instead, it stays high for the parts it did get right (the bottom bun and patty) and then slowly decays as it realizes it's in trouble.
The Result: This teaches the robot that the first half of the attempt was actually good, but the end was bad. It preserves the "partial credit" for the work it did before the mistake.

The authors tested this against four other ways of labeling mistakes (like dropping the score instantly to zero, or scaling the whole score down). They found that their "smooth decay" method was the only one that didn't confuse the robot. It allowed the robot to learn that a failed attempt still had value in the beginning, preventing it from "collapsing" its predictions on successful attempts later.

Two Heads, One Brain

ValueFormer is a small, efficient model that runs alongside the main robot brain. It has a unique "dual-head" design, like a robot with two different types of eyes:

The Smooth Eye ( $V_{mc}$ ): This gives a continuous, gentle progress score (0 to 1). It tells the operator, "We are 60% done," or "We are stalling." It's great for tracking overall progress.
The Sharp Eye ( $V_{bin}$ ): This is a binary alarm. It snaps to "Danger!" the moment a mistake happens. It's designed to catch errors quickly, like when the robot drops the lettuce, so a human can step in before the whole sandwich is ruined.

The paper shows that by training on 1,427 real robot episodes of sandwich-making, this system learned to recognize four distinct patterns:

Clean Success: The score climbs steadily to the top.
Success-with-Retry: The score dips when the robot fumbles (like dropping a patty) but climbs back up once it fixes it.
Early Collapse: The score goes up, then drops and stays low because the robot messed up early.
Stuck-Scratching: The score stays low because the robot is just spinning its wheels.

Real-World Results

When the researchers used this "progress bar" to help the robot learn, the results were promising. By using the robot's own "gut feeling" to weight its training data (telling it to pay more attention to the parts where it was struggling), they improved the robot's success rate from 70% to 85% on a sandwich-making task. They also managed to eliminate a specific bad habit where the robot would keep trying to pick up an ingredient it had already successfully placed (a "repeat-pick" error).

However, the authors are careful to note that while the improvement is real, the sample size (20 sandwiches in the final test) is small, so the results are "suggestive" rather than a guaranteed, solved problem. They also found that the system needs a little bit of time to react—about 5 to 10 seconds—because it looks at a short history of what happened before making a judgment.

Making it Fast Enough to Run

Finally, the team had to make sure this new "progress bar" didn't slow down the robot. Running two brains on one computer chip is hard. They discovered that the main cost wasn't the thinking part, but the "seeing" part (processing the camera images). By optimizing how the images were processed (using a technique called "batching" and a specific math format called "bf16"), they cut the time it took to check the progress by 3 to 5 times. This allowed the system to run smoothly at 2 Hz (twice a second) on the same computer chip as the main robot brain, without causing any lag.

In short, ValueFormer gives robots a way to say, "I'm doing okay, but I just dropped the cheese, and I need help," rather than just silently failing at the end. It turns a blind imitation into a semi-autonomous partner that knows when it's in trouble.

Technical Summary: ValueFormer

Problem Statement

Vision-Language-Action (VLA) policies trained via behavior cloning (e.g., the $\pi_0$ family) often fail silently. Because imitation learning provides no intrinsic notion of progress, a robot making clean progress and one that has stalled or failed can produce identical action streams and visual outputs. While Reinforcement Learning (RL) could theoretically provide a value signal, it is impractical for real-robot tasks involving deformable objects (like food) due to the high cost of data collection and the inability of current simulators to faithfully reproduce physical interactions.

Existing alternatives, such as terminal success/failure bits, suffer from extreme sparsity. A single bit at the end of a two-minute episode cannot inform an operator when a mistake occurred, nor can it provide the dense signal required for advantage estimation or online safety filtering. Furthermore, simple dense labeling schemes (e.g., assigning a value of 0 to all frames in a failed episode) are actively harmful; they teach the model that early-stage observations in a failed rollout are inherently "bad," causing the model to predict failure even on clean successes during deployment.

Methodology

The paper proposes ValueFormer, a compact, policy-agnostic causal transformer that emits two per-frame signals from a single forward pass: a smooth Monte Carlo value ( $V_{mc}$ ) for progress tracking and a sharp binary value ( $V_{bin}$ ) for mistake detection.

1. Stage-Aware, Success-Then-Decay Labeling

The core technical contribution is a novel labeling scheme for failed episodes that addresses the "wrongly shaped label" problem:

Pre-Failure: Frames preceding the failure in a failed episode are labeled identically to successful episodes, following a rising Monte Carlo curve ( $V \approx \gamma^{N-1-k}$ ). This preserves the partial credit for stages completed before the error.
Post-Failure: Instead of a hard drop to zero (a "cliff"), the label decays smoothly from the value at the failure point ( $V_{fail} = V_{succ}(k_{fail}) \cdot \gamma^{k-k_{fail}}$ ). This encodes the intuition that a robot failing late in a task still deserves more credit than one failing early.
Binary Detection: A second target, $V_{bin}$ , is generated from segment-based mistake intervals $(t_{start}, t_{end})$ rather than a single failure point. This allows the model to learn from transient mistakes in otherwise successful rollouts (e.g., a failed grasp that is recovered).

2. Architecture

ValueFormer is a lightweight causal transformer (approx. 3.5M trainable parameters) built on top of a frozen DINOv3 ViT-L/16 backbone.

Input: It processes a short, fixed temporal window (8 seconds, 16 frames) of six views: three full camera images and three Region-of-Interest (ROI) crops (gripper and wrist regions). It also ingests joint state and time features.
Causal Design: The transformer uses a causal mask, ensuring that the value at time $t$ depends only on past and present frames, making it valid for online inference.
Dual-Head Output:
- $V_{mc}$ Head: A 3-layer MLP producing a smooth value in $[0, 1]$ , supervised by the stage-aware Monte Carlo labels.
- $V_{bin}$ Head: A 2-layer MLP producing a sharp probability of failure, supervised by the mistake intervals.

3. Training and Supervision

The model is trained jointly on both heads using Binary Cross-Entropy (BCE). To scale annotation, the authors utilize human-in-the-loop (HIL) intervention flags from teleoperation sessions. When an operator takes over a failing rollout, the interval of intervention is treated as a mistake segment, providing dense, cheap supervision that augments manual annotations.

Key Results

The method was evaluated on a real-robot bimanual sandwich-assembly task with 1,427 episodes (1,249 expert successes, 178 autonomous rollouts).

Labeling Efficacy: An ablation study comparing five labeling shapes demonstrated that the proposed "MC-smooth" (success-then-decay) scheme significantly outperformed alternatives (outcome-scaled, cliff, linear mix, late-diverge). It achieved the lowest validation BCE loss and rollout MAE, while avoiding the "outcome leakage" (predicting failure at $t=0$ based on the final outcome) seen in other schemes.
Qualitative Signatures: ValueFormer successfully reproduced four canonical rollout signatures on held-out data:
1. Smoothly ascending curves for clean successes.
2. Dips and recoveries for successes with retries.
3. Rise-then-drop patterns for early collapses.
4. Flat-low profiles for stuck/scratching episodes.
Failure Detection: By combining manual annotations with intervention-derived segments, the binary head achieved an Average Precision (AP) of 0.82 on held-out failure detection (compared to 0.38 with manual labels alone), detecting 95% of mistakes before the operator intervened.
Policy Improvement (A/B Test): In an on-robot experiment, using the critic to derive per-frame training weights for the VLA policy improved task completion from 70% to 85% (within statistical noise at $n=20$ ) and eliminated a specific "repeat-pick" failure mode that persisted under a flag-only baseline.
Deployment Efficiency: Through batched encoding and BF16 autocast on the frozen encoder, the live serving cost was reduced by 3–5 $\times$ , allowing the critic to run at 2 Hz alongside the policy on a single GPU without stalling the control loop.

Significance and Claims

The paper argues that the label, not the architecture, is the critical bottleneck in learning value functions for behavior cloning. The primary contribution is the demonstration that a dense, correctly shaped, stage-aware label is essential for learning a useful value signal.

Supervision Density: Moving from a terminal bit to a per-frame target is necessary but insufficient; the shape of that target determines success. A dense label with the wrong shape (e.g., a flat zero for all failures) is worse than no label at all because the model fits it faithfully, learning incorrect associations.
Policy Agnosticism: ValueFormer operates alongside a frozen VLA, requiring no modification to the policy's training loop or input format, making it a plug-and-play module for the $\pi_0$ family.
Dual-Target Design: The separation of a smooth value (for progress estimation) and a sharp binary signal (for safety aborts) allows a single model to serve both the operator's need for a "progress bar" and the system's need for an immediate "abort" trigger.

The authors conclude that while the specific application was sandwich assembly, the design pattern—a small, dual-output value head with engineered targets sharing a backbone—offers a generalizable path toward semi-autonomous robots that can detect their own failures before human intervention is required.