Technical Summary: SeeQ: Training Generalist Value Functions for Long-Horizon Robotic Manipulation

Problem Statement

Generalist robot policies, often trained via imitation learning, struggle with complex, long-horizon manipulation tasks. These tasks frequently involve multiple stages or require repeated attempts and deliberation on specific stages before success. In such settings, errors compound over time, and recovery behaviors are underrepresented in expert datasets. While Q-value functions offer a mechanism to steer policies by ranking candidate actions, learning them for long-horizon tasks is challenging. Standard approaches face three primary hurdles:

1. Long Credit-Assignment Horizons: Learning from sparse task-level rewards (success/failure at the very end) requires propagating signals over long sequences, leading to difficult Bellman backups.
2. Data Coverage: Offline datasets often lack sufficient coverage of the diverse state-action pairs encountered during long trajectories, making Temporal-Difference (TD) learning unreliable.
3. Brittleness: Without effective value learning, policies cannot distinguish between actions that make progress and those leading to failure, particularly in precision-heavy or multi-stage tasks.

Existing methods either avoid TD learning (using Monte Carlo returns, which limits policy improvement beyond the data distribution) or rely on TD learning over the full task horizon, which suffers from error compounding.

Methodology: SeeQ (Subtask-elicited Q-functions)

The authors propose SeeQ, a framework for training generalist Q-value functions that shortens the learning horizon by focusing on the currently active subtask rather than the entire task.

Core Architecture and Training

SeeQ utilizes a pretrained Vision-Language Model (VLM) backbone (specifically a 3B PaliGemma model) and introduces a two-stage training process:

1. Generalist Pretraining: The model is pretrained on diverse, open-source robot manipulation datasets (e.g., RoboCOIN) containing subtask annotations. This stage regularizes the model's understanding of task progress and action conditioning across various embodiments.
2. Downstream Finetuning: The model is finetuned on specific target tasks.

Key Technical Innovations

• Subtask-Level Value Prediction: Instead of predicting the return to the final task completion, the Q-function estimates the expected return for the active subtask ($\tilde{l}_t$). This reduces the prediction horizon, making TD learning more stable and effective.
• Autoregressive Subtask Decoding: To eliminate the need for human annotations or external subtask predictors at test time, the Q-function architecture is modified to autoregressively predict the active subtask in natural language before estimating its value.
  • During training, the model is supervised with a next-token prediction loss on ground-truth subtask annotations.
  • During inference, the model generates the subtask text ($\hat{l}_t$) based on the current state and task instruction, then conditions the value prediction on this generated text.
• TD-BoN (Temporal-Difference with Best-of-N) Learning: The training objective combines subtask-level TD learning with a "Best-of-N" backup strategy.
  • Candidate action chunks are sampled from a base policy ($\pi_{base}$).
  • The target value is computed using the action chunk with the highest estimated value among the $N$ candidates.
  • This approach aligns the training backup with the test-time steering procedure and allows the Q-function to evaluate actions better than those observed in the dataset.
• No Bootstrapping Across Boundaries: The TD target does not bootstrap across subtask boundaries. If a subtask ends within the action chunk horizon, the backup term is zeroed out, ensuring the value function strictly reflects the current subtask's progress.

The total loss function combines the TD loss for the subtask value and the next-token prediction loss for the subtask text:
$$\mathcal{L}_{SeeQ}(\theta) = \mathcal{L}_{TD}(\theta) + \lambda_{subtask}\mathcal{L}_{subtask}(\theta)$$

Test-Time Inference

At deployment, SeeQ steers a base policy ($\pi_{base}$) via Best-of-N selection:

1. Given a state $s_t$ and task instruction $l$, the model autoregressively predicts the active subtask $\hat{l}_t$.
2. The base policy proposes $N$ candidate action chunks.
3. The Q-function scores each chunk conditioned on $s_t$, $l$, and the predicted $\hat{l}_t$.
4. The action chunk with the highest score is executed.

Key Contributions

1. Subtask-Level Formulation: The paper introduces a method to recast long-horizon value learning as short-horizon, subtask-level learning, effectively sidestepping the challenges of sparse rewards over long horizons.
2. Language-Conditioned Subtask Inference: A novel architecture that explicitly predicts the active subtask in natural language, removing the need for modular subtask prediction systems or human annotations at test time.
3. Generalist Pretraining Strategy: Demonstrates that pretraining a VLM backbone on diverse robot data is critical for learning robust action-conditioning and reducing overfitting to specific image features during downstream finetuning.
4. Empirical Validation: Extensive evaluation on four real-world bimanual manipulation tasks (shirt-hanging, lid-sealing, grocery-packing, Lego-disassembly) across two robot platforms.

Results

The authors evaluated SeeQ on four real-world tasks involving deformable objects, precise alignment, and multi-stage coordination.

• Policy Steering Performance: SeeQ substantially improved the success rates of base policies across all tasks.
  • Shirt-hang: Improved from 10/24 (41.7%) to 22/24 (91.7%).
  • Lid-sealing: Improved from 10/24 (41.7%) to 15/24 (62.5%).
  • Grocery-packing: Improved from 9/24 (37.5%) to 17/24 (70.8%).
  • Lego-disassembly: Improved from 5/24 (20.8%) to 10/24 (41.7%).
  • Overall, the average success rate increased from 35.4% to 66.7% (a 1.88x improvement).
• Ablation Studies:
  • Pretraining: Removing robot data pretraining caused a significant drop in performance (from 22/24 to 8/24 on shirt-hang), highlighting the necessity of diverse data for generalization.
  • Subtask Conditioning: Explicitly decoding and conditioning on the subtask was critical. Removing subtask prediction led to performance below the base policy (8/24), while adding prediction without conditioning improved it to 15/24. Full SeeQ achieved 22/24.
  • Comparison to Baselines: SeeQ outperformed task-level Monte Carlo regression, task-level TD learning, and subtask-level SARSA (which uses dataset actions for backups rather than Best-of-N sampling).

Significance and Claims

The paper claims that subtask-level values provide a more effective learning target than sparse, long-horizon success signals when training generalist Q-functions. By leveraging the natural decomposition of manipulation tasks into intermediate milestones, SeeQ enables effective TD learning without the error compounding typically associated with long horizons.

The authors emphasize that their approach allows for generalist value learning that can be applied to new tasks with minimal finetuning, provided the tasks can be decomposed into subtasks. The explicit use of language to predict subtasks aligns the value estimation with the VLM's text-generation capabilities, improving robustness near subtask transitions.

The work suggests that combining short-horizon learning objectives with language-structured inference is a viable path toward more robust, long-horizon robotic manipulation, particularly when leveraging large-scale pretraining on diverse robot data. The authors acknowledge limitations, such as the ambiguity of subtask boundaries and the reliance on the quality of the base policy's candidate actions, but posit that their framework offers a significant step forward in making value functions practical for complex real-world robotics.