Technical Summary: HALO – History-Aware Visuomotor Policy for Long-Horizon Robotic Imitation Learning

1. Problem Statement

General-purpose robots operating in partially observable environments (e.g., households) must perform long-horizon manipulation tasks that require recalling information no longer present in current sensory inputs. Examples include remembering object locations, tracking numerical counts, recalling event timing, or understanding relational changes between objects.

Standard reactive visuomotor policies fail in these scenarios because they lack memory. While existing approaches attempt to address this through hand-designed memory representations, modality-specific retrieval rules, or learned compression (e.g., latent state summaries), they often struggle with generalizability. They rely on task-specific assumptions that limit scalability across diverse manipulation tasks.

Directly applying attention-based memory retrieval (e.g., Transformers) to offline imitation learning introduces two fundamental challenges:

1. Spurious Correlations: The policy may learn to attend to past observations that correlate with expert behavior in the training data but are not causally relevant to task execution, leading to poor generalization at test time.
2. Error Accumulation and Model Drift: In closed-loop execution, small prediction errors compound over time. Conditioning on large amounts of noisy historical information can amplify these errors, leading to cascading failures and model drift over long horizons.

2. Methodology: HALO

The authors propose HALO (History-Aware visuomotor policy for LOng-horizon robotic imitation learning), a framework designed to enable end-to-end execution of tasks spanning up to eight minutes by retrieving relevant information from history. The core insight is that while attention provides a general retrieval mechanism, it must be guided toward task-relevant information and constrained to avoid conditioning on irrelevant or noisy history.

A. Architecture Overview

HALO parameterizes a visuomotor policy $\pi_\theta(a_t | \tau_t, l)$ with three main components:

1. Encoders: Modality-specific encoders ($g_{obs}$, $g_{act}$) map heterogeneous inputs (RGB, proprioception, past actions) into a common latent space.
2. Policy Backbone: A Transformer-based backbone ($f_\theta$) performs memory retrieval over the encoded history $M_t$ and processes the information.
3. Prediction Heads: A task-specific action head ($h_{act}$) predicts low-level control commands, and a text prediction head ($h_{text}$) provides auxiliary supervision.

B. Key Mechanisms

1. Distilling VLM Priors via Video Question-Answering (VQA)

To mitigate spurious correlations, HALO leverages Vision-Language Models (VLMs) to distill priors about which historical information is task-relevant.

• Mechanism: The policy is co-trained to both imitate expert actions and answer questions about the trajectory history.
• VQA Generation: A multi-stage pipeline generates high-quality VQA pairs from long-horizon trajectories. This involves:
  1. Trajectory-to-text summarization: Combining object detectors and VLMs to create textual descriptions of scene elements and activities.
  2. Question synthesis: An LLM generates questions probing specific memory requirements (e.g., object locations, event ordering) tied to specific frames.
  3. Verification: An LLM verifier filters out incorrect or weakly relevant pairs.
• Training Objective: The policy minimizes a joint loss: $L(\theta) = L_{IL}(\theta) + \lambda L_{VQA}(\theta)$. The VQA objective forces the retrieval module to attend to specific parts of the history where task-relevant events occurred, providing direct supervision on what to retrieve.

2. Top-k Attention Sparsification

To reduce the impact of accumulated errors during closed-loop control, HALO restricts the policy to condition only on a sparse subset of stored information.

• Mechanism: Instead of attending to all history, the policy computes query-key similarities and selects only the top-$k$ most relevant memory entries.
• Training: A straight-through estimator is used to handle the non-differentiable top-k operation. Gradients reinforce the selection of entries that improve action prediction or VQA accuracy, while down-weighting irrelevant entries.
• Benefit: This limits the influence of noisy information, reducing model drift and mitigating cascading failures.

3. Key Contributions

• General Memory Retrieval: Introduces a learned, end-to-end memory retrieval mechanism that avoids task-specific assumptions or hand-designed rules.
• VQA-Guided Retrieval: Proposes using automatically generated Video Question-Answering supervision to steer attention toward task-relevant information, effectively transferring VLM priors to the visuomotor policy.
• Sparse Attention for Robustness: Demonstrates that top-k sparsification effectively reduces model drift in long-horizon closed-loop control by filtering out accumulated noise.
• Comprehensive Evaluation: Validates the approach on the REMEMBENCH benchmark and five real-world tasks across two robotic platforms (fixed-base and mobile manipulators).

4. Experimental Results

Simulation (REMEMBENCH)

Evaluated on tasks requiring spatial, relational, numerical, and event-time memory:

• Overall Performance: HALO achieved an average success rate of 41%, outperforming the Standard Transformer (22%), SAM2Act++ (20%), ReMemBer (18%), and Hand-Designed Features (29%).
• Ablation Studies:
  • Removing VQA supervision reduced average success by 10% (to 31%), highlighting the importance of VLM priors for identifying relevant history.
  • Removing top-k sparsification (using full attention) reduced performance by 9%, confirming its role in reducing drift.
  • Co-training VQA and action prediction outperformed a pretrain-then-finetune approach by 20% (64% vs. 44%), suggesting that joint training is crucial for shaping retrieval effectively.
• Comparisons: HALO outperformed alternative sparsification methods (window, strided, hierarchical, gated attention) and non-associative memory models (LSTM, Mamba, Transformer-XL).

Real-World Evaluation

Evaluated on five real-world tasks (including human-robot collaboration):

• HALO achieved an average success rate of 55%, compared to 36% for the Standard Transformer baseline.
• Notable improvements were observed in challenging long-horizon tasks, such as heating a stove for up to eight minutes and storing multiple objects.
• Analysis of the "Retrieve Object" task showed HALO reduced manipulation failures by 8% and memory failures by 25% compared to the baseline.

5. Significance and Claims

The paper claims that HALO provides a general solution for long-horizon control in partially observable environments by learning to retrieve task-relevant information directly from data.

• Versatility: Unlike prior methods that rely on fixed representations (e.g., object-centric) or text summaries, HALO adapts to diverse information types (spatial, numerical, temporal) without manual engineering.
• Necessity of Grounding: The authors emphasize that while VLM priors are useful for selecting relevant information, they are insufficient on their own for low-level control (achieving only 18% success without action grounding). The combination of VLM-guided retrieval and action-grounded imitation is critical.
• Robustness: The work demonstrates that selective retrieval via top-k sparsification is a practical and effective strategy to mitigate the compounding errors inherent in long-horizon closed-loop execution.

The authors conclude that their approach enables reliable control over extended time horizons (up to eight minutes) by balancing the need for rich historical context with the necessity of filtering out noise and irrelevant correlations. Future work is proposed to explore active querying strategies and cross-episodic memory learning.