🤖 AI

Offline Policy Evaluation for Manipulation Policies via Discounted Liveness Formulation

This paper proposes a novel offline policy evaluation framework for robotic manipulation that utilizes a discounted liveness-based Bellman operator to effectively handle sparse rewards, non-monotonic task progression, and finite-horizon truncation bias, thereby outperforming classical methods in accurately reflecting task progress.

Original authors: Hao Wang, Joshua Bowden, Colton Crosby, Somil Bansal

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hao Wang, Joshua Bowden, Colton Crosby, Somil Bansal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a robot learn to fold a towel or pick up a bowl. Sometimes, the robot succeeds. Sometimes, it drops the bowl, slips, or runs out of time before finishing.

In the world of robotics, we need a way to grade these attempts. We want to know: "How good is this robot's strategy?" The problem is that the robot only gets a grade at the very end of the attempt: Pass (Task Done) or Fail (Time's Up). It doesn't get a score for every single step along the way.

This paper introduces a new, smarter way to grade these robots, specifically when they are allowed to keep trying even after making mistakes.

The Problem: The "Time-Out" Trap

Imagine a student taking a test. If they run out of time, the teacher marks the whole test as "Incomplete." But what if the student was actually on the right track and just needed two more minutes? Or what if they got stuck on one hard question, took a break, solved an easier one, and then came back to the hard one?

In robotics, this is called truncation bias. Because the robot's "test" (the episode) has a hard time limit, standard grading methods assume that if the robot stopped because time ran out, it was a total failure. They don't realize the robot might have been almost there or was in the middle of a recovery. This makes the robot look worse than it actually is.

The Solution: The "Liveness" Score

The authors propose a new way to think about the score. Instead of asking, "How much reward did you get?", they ask, "How close are you to being 'alive' (successful)?"

They treat the task like a game of "Hot or Cold":

Goal State (Success): The robot is "hot." The score is very low (like -1).
Failure/Time-out: The robot is "cold." The score is high (like 1).
In Between: The score tells you how many steps it should take to get to the goal.

The clever part is how they handle the "Cold" moments.

The "Bootstrap" Trick: Learning from Near-Misses

Here is the magic ingredient: Bootstrapping.

Imagine you are watching a robot try to pick up a bowl.

Attempt 1: The robot picks up the bowl, drops it, picks it up again, and succeeds.
Attempt 2: The robot picks up the bowl, drops it, and then the timer runs out.

A standard grader would say Attempt 2 is a total failure. But our new method looks at Attempt 2 and says, "Wait a minute! The robot dropped the bowl in Attempt 2, just like it did in Attempt 1. But in Attempt 1, the robot recovered from that exact same drop and finished the task."

So, the method says: "Even though Attempt 2 ran out of time, the robot was in a state it knows how to recover from. Let's give it a better score."

This is the Bootstrap Mechanism. It looks for moments in "failed" attempts that look exactly like moments in "successful" attempts. If it finds a match, it upgrades the score of the failed attempt, realizing that the robot wasn't doomed; it was just running out of time.

What They Found

The authors tested this on three different scenarios:

Picking up a bowl (Simulation): The robot dropped the bowl, recovered, and finished. The new method correctly gave the robot credit for the recovery, while old methods thought it was a failure.
Putting a square peg in a hole (Simulation): This was harder because the robot didn't have a "recovery" strategy. If it dropped the peg, it was truly stuck. The new method was still good at spotting progress, but it couldn't magically fix the fact that the robot couldn't recover from a bad drop.
Folding a towel (Real Robot): A human was controlling the robot. Humans often struggle, drop the cloth, and try again. The new method was excellent at realizing that even when the human struggled, they were still making progress toward the goal, whereas old methods just saw "struggling = failure."

The Trade-off

The paper admits a small downside. Because the method is so good at giving credit for "almost there" moments, it sometimes gets a little too optimistic. It might think a failed attempt was actually a success because it looked similar to a real success. However, the authors argue that this is a fair trade: it's better to be slightly too optimistic about a robot's ability to recover than to unfairly punish it for running out of time.

In a Nutshell

This paper gives robots a "second chance" in their grading system. Instead of failing a robot just because the clock ran out, the new method looks at the robot's history. If the robot has shown it can recover from a specific mistake before, the method gives it credit for being on the right track, even if the current attempt didn't finish in time. It turns a simple "Pass/Fail" grade into a nuanced map of how close the robot really is to success.

Technical Summary: Offline Policy Evaluation for Manipulation Policies via Discounted Liveness Formulation

Problem Statement

The paper addresses the challenge of offline policy evaluation (OPE) for robotic manipulation policies. While policy evaluation is critical for developing and deploying robotic systems, standard Reinforcement Learning (RL) methods face significant hurdles in real-world manipulation scenarios due to three specific properties:

Sparse Rewards: Rewards are typically binary (task completion vs. failure) rather than dense, making it difficult to learn intermediate progress.
Finite-Horizon Truncation: Evaluation rollouts are often terminated by manual time-outs. In offline data, it is impossible to distinguish between a policy that has truly failed and one that simply ran out of time but could have succeeded given more duration. This introduces truncation bias, leading standard methods (like TD(0) and Monte Carlo) to systematically underestimate policy performance.
Non-Monotonic Progression: Manipulation policies often exhibit recovery behaviors (e.g., dropping an object and picking it up again). This causes task progression to be non-monotonic, violating assumptions made by cumulative reward formulations that expect a steady approach toward the goal.

Methodology

The authors propose a framework that reformulates offline policy evaluation as a discounted liveness problem. Instead of estimating the expected cumulative return, the method learns a value function that represents the "semantic distance" to task completion.

1. Discounted Liveness Formulation

The authors modify the standard Markov Decision Process (MDP) by replacing the reward function $R$ with a target function $l(s)$ , which encodes the distance to a target set $L$ (goal states).

Target Function: $l(s) = -1$ if $s$ is a goal state, and $l(s) = 1$ otherwise.
Value Definition: The value of a state $s$ is defined as the expectation of the minimum target value achieved over all possible future trajectories:
$V_\pi(s) = \mathbb{E} \left[ \min_{k} l(\xi(k)) \right]$
Conservative Estimate: To avoid over-optimism, the authors derive a conservative (pessimistic) fixed-point Bellman operator. They introduce a discount factor $\gamma$ to induce a contraction mapping, resulting in the update rule:
$\tilde{V}_\pi(s) = (1 - \gamma) + \gamma \min \{ l(s), \tilde{V}_\pi(s') \}$
This formulation ensures that if a trajectory encounters a failure (value 1), the value propagates backward, but if it encounters a goal (value -1), the value reflects the steps to that goal.

2. Two-Stage Bootstrapping Framework

To address the lack of dense rewards and the issue of truncation bias, the authors introduce a two-stage algorithm:

Stage 1 (Anchor Computation): The algorithm identifies "unambiguous" states—specifically, all states in episodes that successfully reach a goal. It computes value estimates for these states and their descendants using the Bellman operator. These values serve as "anchors" grounded in observed success.
Stage 2 (Bootstrapping): The target function $l(s)$ $l (s)$ is refined using the value function $V_\pi$ $V_{π}$ from Stage 1. For states in timed-out (failure) episodes, if a state (or a similar state in the latent space) overlaps with a state from a successful episode, its target value is updated from the default pessimistic value (1) to the optimistic value derived from the successful trajectory.
- The update rule becomes: $\tilde{V}_\pi(s) = (1 - \gamma) + \gamma \min \{ V_\pi(s), \tilde{V}_\pi(s') \}$ .
- This mechanism effectively propagates success signals backward through truncated episodes, overwriting the assumption that time-outs imply inevitable failure.

Key Contributions

Liveness-Based Formulation: A novel reformulation of OPE for sparse-reward manipulation tasks that interprets value as a measure of task progression (liveness) rather than cumulative return.
Truncation Bias Mitigation: A two-stage, bootstrapped evaluation algorithm that leverages observed successes to correct the pessimistic bias inherent in finite-horizon offline data.
Theoretical Guarantees: Proof that the proposed discounted Bellman operator induces a contraction mapping, ensuring the existence of a unique fixed-point value function.

Experimental Results

The method was evaluated on three case studies using datasets generated by Vision-Language-Action (VLA) models, diffusion policies, and human demonstrations:

LIBERO-Spatial (Pick and Place): A simulated task with a VLA policy.
Square Peg Insertion: A simulated task with a diffusion policy.
Cloth Folding: A hardware task on a Franka Panda arm using human teleoperation data.

Baselines: The method was compared against TD(0), Monte Carlo (MC), Distributional MC (MC-D), and an ablation of their own method without bootstrapping (Ours-NB).

Metrics:

Success Metric: Percentage of frames in successful episodes where the estimated steps-to-goal are less than the actual episode length.
Failure Metric: Percentage of frames in failed episodes where the estimated steps-to-goal exceed a standard task horizon.
Composite Metric: The arithmetic mean of the above.

Findings:

Performance: The proposed method achieved the best performance in the Success Metric and Composite Metric across all three tasks, significantly outperforming classical baselines.
Truncation Bias: The method substantially reduced truncation bias, correctly identifying that many states in timed-out episodes were actually close to success (recoverable) rather than failures.
Failure Metric Trade-off: The method performed slightly worse on the Failure Metric compared to some baselines. The authors attribute this to the bootstrapping mechanism being "optimistic" for states in timed-out episodes that visually resemble successful states, potentially misclassifying some true failures as recoverable.
Robustness: The method remained effective even with smaller dataset sizes (down to 50 episodes) and was largely insensitive to the choice of image encoder (SigLIP2, CLIP, DINOv2).

Significance and Claims

The paper claims that by reframing policy evaluation as a liveness problem and utilizing a bootstrapping mechanism, it is possible to learn value functions that accurately reflect non-monotonic task progression in manipulation tasks. The primary significance lies in the ability to distinguish between policy failure and episode truncation, a critical capability for reliable offline evaluation in real-world robotics where time-outs are common.

The authors note that their approach is particularly effective for policies that exhibit recovery behaviors. However, they acknowledge limitations:

The bootstrapping mechanism can lead to over-optimism if the state overlap between successful and failed episodes is learned incorrectly (e.g., in tasks with poor visual cues like the Square task).
The method relies heavily on the dataset distribution and may not generalize well outside of it.
Performance degrades for policies that do not exhibit recovery behaviors, as the bootstrapping mechanism has fewer "anchors" to propagate from.

The work concludes that while the method introduces a small trade-off in failure detection accuracy, the substantial improvement in capturing successful task progression makes it a superior tool for evaluating manipulation policies under sparse rewards and finite horizons.