Technical Summary: History-Aware VErifier (HAVE)

Problem Statement

Robots operating in the real world frequently encounter visually ambiguous objects where identical appearances mask vastly different physical behaviors (e.g., a door that requires pushing versus pulling, or objects with unknown internal mass distributions). In such scenarios, the true state of the environment is partially observable, making it a Partially Observable Markov Decision Process (POMDP).

The core challenge is that standard generative policies, even when conditioned on action history, struggle to disambiguate these cases. Theoretical and empirical evidence suggests that training a single generative policy to imitate heterogeneous expert datasets (where the ground-truth solution varies for the same observation) leads to suboptimal performance. The authors argue that relying solely on a history-conditioned generator fails to effectively leverage past interactions to resolve uncertainty in real-time.

Methodology: HAVE Framework

The authors propose HAVE (History-Aware VErifier), a framework that explicitly decouples action generation from action verification. The system operates on a generation-verification paradigm:

1. Unconditional Generator ($\pi_G$): A diffusion-based model generates a batch of $N$ candidate actions based on the current observation. Crucially, this generator is unconditional regarding history; it proposes diverse candidates without being biased by past failures or successes during the generation phase.
2. History-Aware Verifier ($V$): A neural network evaluates the proposed candidates by reasoning about the full history of interactions (past actions and their resulting observations). It outputs a scalar score (estimated reward) for each candidate.
3. Selection: The system selects and executes the action with the highest estimated score from the verifier.

Network Architecture

The verifier employs a specific architecture designed to relate current proposals to historical data:

• Encoding: Actions are represented as dense action fields (point clouds with action vectors) and encoded via PointNet++. Observations are encoded by computing "Observation Flow" (the movement of points between consecutive frames) and encoding this flow.
• Attention Mechanism: The core of the verifier uses an explicit attention layer. The current action proposal serves as the Query (Q), while embeddings of historical actions and observation flows serve as Keys (K) and Values (V). This allows the model to retrieve relevant past experiences similar to the current proposal.
• Unconditional Branch: To handle cases where no similar history exists, the architecture includes an unconditional evaluation branch that scores the action without historical context, which is integrated into the final attention mechanism.
• Training Loss: The model is trained using three loss components:
  • Final Score Loss: Supervises the final predicted score for the selected action.
  • History Score Loss: Supervises the scores of historical action results.
  • Unconditional Score Loss: Supervises the unconditional evaluation branch.

Theoretical Motivation

The paper provides theoretical justification for the generation-verification split:

• Generation-Verification Gap: Citing Setlur et al., the authors note that in settings with heterogeneous solution traces, a generative policy suffers a reward gap that contracts at a rate of $O(\sqrt{1/n})$, whereas a verifier trained to fit reward terms contracts at a faster rate of $O(e^{-H/n})$ (where $H$ is horizon length).
• Selection Benefit: The authors prove that if a verifier has an accuracy greater than 50% ($p_V > 0.5$) and the generator has a non-zero probability of sampling a good action, selecting the best action from $N$ samples using the verifier yields a strictly higher expected reward than sampling a single action from the generator.

Experimental Results

The method was evaluated in three distinct environments involving fundamental ambiguities, in both simulation and real-world settings:

1. Articulated Objects (Simulation):

   • Task: Opening various articulated objects (PartNet-Mobility) from a closed state.
   • Results: HAVE reduced the failure rate by approximately 6x compared to FlowBot3D (from 13% to 2%) and 10x compared to a history-conditioned generator without a verifier.
   • Ablations: Using dense action fields and explicit observation flow extraction proved superior to sparse representations and raw point cloud sequences.

2. Multi-modal Doors (Simulation & Real World):

   • Task: Determining the correct opening direction (push/pull, left/right) for doors with identical geometry but different joint configurations.
   • Results: HAVE significantly outperformed baselines in both failure rate and the number of steps required to open the door. In real-world experiments with a Franka Emika Panda robot, HAVE demonstrated higher success rates and efficiency across all four door modes.

3. Uneven Object Pick-up (Simulation):

   • Task: Lifting rods and objects with unknown, uneven mass distributions.
   • Results: HAVE reduced failure rates on both ambiguous rod datasets and unseen object datasets compared to baselines, demonstrating the ability to infer the center of mass through trial-and-error interactions.

Key Contributions

1. Novel Verifier Structure: A history-aware verifier that explicitly reasons about past interactions to disambiguate uncertain scenarios, decoupling this reasoning from the action generation process.
2. Theoretical Analysis: A formal demonstration that using a verifier to select from generator proposals improves expected action quality, provided the verifier is better than random chance.
3. Empirical Validation: Comprehensive experiments across simulated and real-world environments (articulated objects, ambiguous doors, uneven mass objects) showing significant improvements over state-of-the-art baselines and ablation studies confirming the efficacy of specific design choices (e.g., dense action fields, explicit attention).

Significance and Limitations

The paper claims that HAVE offers a robust solution for online disambiguation in ambiguous, multi-modal manipulation tasks. By combining a generative policy with a history-aware verifier, the system can adapt its strategy based on trial-and-error outcomes, overcoming the limitations of purely generative approaches.

The authors acknowledge limitations:

• The system's performance is strictly bounded by the expressiveness of the generator; if the generator fails to propose a viable action, the verifier cannot compensate.
• Current training relies on privileged knowledge (ground-truth actions and scores) obtained from simulation, which may not be directly available in diverse real-world scenarios.
• Future work aims to explore how verifiers can actively improve generator quality and how to learn scores without privileged supervision.