💻 computer science

Decoding Task Progress from VLA Representations

This paper demonstrates that task progress in Vision-Language-Action (VLAs) is linearly readable from their internal activations, enabling the use of simple probes for effective, label-free out-of-distribution detection and runtime monitoring of deployed robotic policies.

Original authors: Atiksh Bhardwaj, Edward Weiyi Duan, Prithwish Dan, Wei-Chiu Ma, Preston Culbertson

Published 2026-08-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Atiksh Bhardwaj, Edward Weiyi Duan, Prithwish Dan, Wei-Chiu Ma, Preston Culbertson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where robots aren't just clumsy arms that repeat the same motion over and over, but clever helpers that can understand your voice, see the mess on your counter, and figure out how to clean it up. This is the dream of "Vision-Language-Action" (VLA) models. Think of them as a robot's brain that combines a super-smart camera (vision), a language translator that understands your commands (language), and a set of muscles that actually move (action). Scientists have been building these brains using massive pre-trained models—like giving the robot a library of every book and movie ever made so it already knows what a "cup" or a "vase" looks like before it ever touches a real one.

But here's the tricky part: once we fine-tune these robots to do specific jobs, like putting a banana on a plate, they sometimes start acting weird. They might get confused by a slight change in lighting, or they might keep trying to grab a banana that isn't there, failing silently without us knowing why. We need a way to peek inside the robot's brain while it's working to see if it's actually making progress or if it's just spinning its wheels. This is where the idea of "interpretability" comes in. It's like having a dashboard that tells you not just what the robot is doing, but what it thinks is happening. If we can read the robot's internal thoughts, we can spot when it's stuck and fix it before it breaks something.

In this paper, a team of researchers decided to see if they could read a very specific thought from inside a robot's brain: how much of the task is left to do? They call this "task progress." Imagine you are eating a pizza. You know you are halfway done when you've eaten half the slices. The researchers wanted to know if the robot's internal "brain waves" (which are just numbers inside its computer) naturally contain a signal that says, "Hey, I'm 50% done!" or "Oh no, I'm only 10% done!"

They tested this on a robot model called $\pi0.5$ , which is like a high-tech robot brain built on top of a famous language and image model called PaliGemma. They didn't teach the robot to calculate progress; they just asked, "If we look at the numbers inside the robot's brain right now, can we guess how much time is left in the task?"

The Big Discovery: The Robot Knows (But Won't Listen)

The answer was a resounding yes. The researchers found that task progress is written clearly in the robot's brain, almost like a line drawn on a graph. If you take a simple math tool (called a "linear probe") and look at the numbers in the first few layers of the robot's brain, it can tell you exactly how much of the task remains. This is amazing because it means the robot naturally understands the concept of "time left" without anyone explicitly teaching it to count down.

Even cooler, they found that this "progress signal" was already there before the robot was ever trained on a specific robot task. It was baked into the big pre-trained brain (PaliGemma) just from reading books and looking at pictures. It's as if the robot learned the concept of "finishing a story" just by reading stories, and it applies that same feeling to moving a cup from a table to a fridge.

The "Magic Wand" Test: Can We Control It?

Here is where things get interesting. The researchers wondered: "If we know the robot is thinking about progress, can we poke its brain to make it think it's further along than it really is?" They tried to "steer" the robot by injecting a signal that said, "You are 20% closer to the goal!"

The result? Nope. The robot didn't listen. Even though the researchers could read the progress signal clearly, they couldn't change the robot's behavior by faking that signal. It's like looking at a car's speedometer and seeing it says "60 mph," then trying to push the needle to "100 mph" with your finger. The needle might move, but the car doesn't actually go faster. The robot's brain knows the progress, but that knowledge doesn't seem to be the main switch that controls its muscles. This suggests a gap: the robot has a rich internal map of where it is, but it doesn't necessarily use that map to decide what to do next in the way we hoped.

The "Stuck Robot" Alarm

So, if we can't control the robot with this signal, is it useless? Not at all! The researchers found a super practical use for it: spotting when the robot is stuck.

Imagine you are watching a robot try to put a rose in a vase. If the robot is working normally, the "progress signal" should go down smoothly, like a timer ticking down. But if the robot gets confused—maybe the vase is in a weird spot, or the lighting changes—the robot might stop making progress. It might keep trying the same failed move over and over.

The researchers built a simple alarm system using this progress signal. They told the system: "If the progress signal stops moving down, sound the alarm!" They tested this against other, more complicated methods that require the robot to be trained on examples of failure. Their simple "progress alarm" worked just as well, and sometimes even better, at spotting when the robot was failing. It didn't need to be taught what failure looked like; it just knew that if the robot wasn't getting closer to the goal, something was wrong.

What This Means for the Future

The paper suggests that these robot brains are full of hidden, easy-to-read signals about what they are doing. We can use these signals as a lightweight, cheap way to monitor robots in the real world. If a robot gets stuck, we don't need a complex AI to figure out why; we can just check its internal "progress clock." If the clock stops, we know it's time to step in.

While the researchers couldn't use these signals to magically force the robot to succeed (the "steering" part didn't work), they proved that the robot does have a clear internal sense of progress. This gives us a new tool to keep our future robot helpers safe, honest, and on track, ensuring they don't just pretend to work while actually doing nothing.

Technical Summary: Decoding Task Progress from VLA Representations

Problem Statement

Vision-Language-Action (VLA) models are emerging as general-purpose policies for robot manipulation, leveraging pretrained Vision-Language Model (VLM) backbones to support open-vocabulary tasks. However, deploying these models often requires task-specific fine-tuning, which can degrade generalization, reduce language sensitivity, and increase brittleness to perceptual perturbations. A critical gap exists in the ability to monitor the internal state of these models at runtime. While existing interpretability tools can decode what a policy "knows" about the world (e.g., object states), they do not currently address where the policy stands relative to its specific task. Specifically, there is a lack of tools to monitor task progress—the normalized time remaining in a trajectory—internally, which is essential for detecting stalled execution or out-of-distribution (OOD) failures without requiring external reward signals or OOD labels.

Methodology

The authors investigate whether task progress is linearly readable from the internal activations of the VLA model $\pi_{0.5}$ (based on the PaliGemma backbone). The methodology is structured around three core concepts: decodability and steerability, and the application of these signals to OOD detection.

1. Feature Definition and Extraction

Task Progress ( $\tau$ ): Defined as the normalized fraction of a trajectory remaining until completion ( $1 - t/T$ ), where $t$ is the current timestep and $T$ is the total trajectory length. This label is derived directly from offline expert demonstrations without additional annotation.
Embedding Extraction: The authors extract residual stream activations ( $z_i$ ) from the transformer layers of $\pi_{0.5}$ for each observation in the dataset.
Linear Probing: A linear probe $\phi(z) = w^\top z + b$ is trained to predict $\tau$ from the activations.

2. Formalizing Representations

The paper formalizes three distinct levels of feature representation:

Weak Decodability: The ability to extract a linear signal correlated with $\tau$ from in-distribution activations.
Strong Decodability: The ability of a single probe (without retraining) to generalize to counterfactually transformed inputs (e.g., swapping the target noun in the language prompt). This tests if the signal depends on the correct semantic inputs rather than spurious correlations.
Steerability: The ability to intervene on the activations (by shifting them along the probe's weight vector) to causally alter the model's behavior (e.g., forcing the model to act as if it has made more progress).

3. OOD Detection Mechanism

A runtime detector is constructed using the progress probe. It compares the predicted progress ( $\tau_{pred}$ ) against the expected progress based on the elapsed time ( $1 - t/E[T|\ell]$ ). A significant residual ( $V_\tau(t) > \delta$ ) indicates that the policy is stalled, flagging the state as OOD. This method requires no OOD labels during training.

Key Contributions

Definitions: The authors introduce formal definitions for weak decodability, strong decodability, and steerability in the context of generative models, distinguishing between the existence of a signal, its dependence on correct inputs, and its causal effect on outputs.
Decodability Findings: They demonstrate that task progress is both weakly and strongly decodable from the residual stream of $\pi_{0.5}$ $π_{0.5}$ .
- The signal is present in the pretrained PaliGemma backbone before any robot-specific fine-tuning, suggesting it emerges from large-scale robotic pre-training.
- A probe trained with language counterfactuals (swapping task nouns) correctly adjusts its output, proving the signal is language-grounded and not just a function of time or visual features.
Steerability Limitation: Despite being decodable, the task progress signal is not steerable. Injecting a "boost" in progress into the activations does not propagate through the network to alter the policy's future actions or successfully steer the model toward the counterfactual behavior.
OOD Detection: The progress probe serves as a label-free OOD detector. It detects stalled progress when the policy encounters perturbations (noise, blur, occlusion) or unseen tasks.

Results

Emergence of Signal: The progress feature is observable across all tested models, including the raw PaliGemma backbone, base $\pi_{0.5}$ , and fine-tuned variants. However, the $\pi_{0.x}$ variants (trained on robotic data) significantly outperform the raw VLM backbone, indicating the feature is established during robotic pre-training.
Counterfactual Robustness: A probe trained with language counterfactuals achieves strong decodability ( $R^2 \approx 0.33$ on in-distribution data due to noisy proxies, but correctly tracks $\tau=1.0$ under swapped instructions), whereas a standard probe fails to respond to language changes.
Steerability Failure: Experiments injecting a progress boost ( $\tau_{boost} = 0.2$ ) at layer 0 show that the predicted progress does not change in subsequent layers, and the resulting actions do not align with the counterfactual target. The injected actions remain distant from the ground-truth future actions.
OOD Detection Performance:
- Cross-Task Generalization: The progress-based detector ( $V_\tau$ ) achieves an AUROC of 0.833 on unseen tasks (compared to 0.796 on seen tasks), demonstrating robustness without retraining. It outperforms density-based methods (Mahalanobis, VAE) on unseen tasks and is competitive with supervised baselines (SAFE-MLP) without requiring OOD supervision.
- Cross-Perturbation Generalization: When trained only on in-distribution data and tested on unseen noise modes (e.g., trained on noise A/B, tested on C/D), $V_\tau$ achieves a per-episode AUROC of 0.960, outperforming supervised detectors that were trained on specific noise types.

Significance and Claims

The paper claims that VLAs possess rich, linearly readable internal representations of semantic quantities like task progress. This finding suggests that:

Monitoring is Feasible: Learning to read these signals offers a lightweight, interpretable path toward monitoring deployed visuomotor policies without the need for complex external models or labeled failure data.
Pre-training is Key: The emergence of task progress in the pretrained backbone indicates that large-scale robotic pre-training instills a structured understanding of task completion that persists through fine-tuning.
Observability vs. Controllability: There is a distinct gap between what a policy represents (observable progress) and what it acts upon (steerability). The inability to steer the model via progress injection highlights that the policy does not use this specific internal signal as a direct control variable for decision-making.

The authors position this work as a step toward instrumenting deployed VLAs, providing a tool to detect silent failures (stalled progress) that is robust to both task variations and environmental perturbations.