💻 computer science

HAMLET: Switch your Vision-Language-Action Model into a History-Aware Policy

The paper introduces HAMLET, a scalable framework that enhances Vision-Language-Action models by integrating time-contrastive moment tokens and a lightweight memory module to effectively leverage historical context, significantly improving robotic manipulation performance on long-horizon and history-dependent tasks.

Original authors: Myungkyu Koo, Daewon Choi, Taeyoung Kim, Kyungmin Lee, Changyeon Kim, Younggyo Seo, Jinwoo Shin

Published 2026-04-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Myungkyu Koo, Daewon Choi, Taeyoung Kim, Kyungmin Lee, Changyeon Kim, Younggyo Seo, Jinwoo Shin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to do chores, like making a sandwich or tidying up a room. You give it a camera and a voice command: "Put the apple in the bowl."

Most current robot brains (called Vision-Language-Action models or VLAs) work a bit like a goldfish with a very short memory. They only look at the exact picture they see right this second. They don't remember what happened a second ago.

This causes problems. If the robot picks up the apple, puts it down, and then the camera angle shifts so the apple is hidden behind a cup, the robot panics. It sees a cup and a table, but no apple. It forgets, "Oh wait, I just held that apple!" and might try to pick up the cup instead, or just freeze.

The paper you shared introduces a new system called HAMLET. Think of HAMLET as giving the robot a smart diary and a highlighter pen.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Goldfish" Robot

Current robots are like people trying to solve a puzzle while wearing blinders. They only see the current frame.

The Analogy: Imagine playing a game of "Simon Says" but you can only hear the current instruction and have no memory of the previous ones. If Simon says, "Touch your nose, then touch your toes," and you only hear "Touch your toes," you might touch your toes while your nose is still in the air, or you might forget you were supposed to touch your nose first.
The Consequence: Robots fail at long tasks (like "pick up the cup, move it to the table, then stack a book on it") because they forget the intermediate steps.

2. The Solution: HAMLET (The Smart Diary)

HAMLET doesn't force the robot to relearn everything from scratch (which is slow and expensive). Instead, it adds two clever tools to the existing robot brain:

A. Moment Tokens (The "Highlighter")

Every time the robot takes a picture, it creates a tiny, compact note called a Moment Token.

How it works: Instead of saving the whole blurry, high-definition photo (which takes up too much space), the robot uses a special "highlighter" to mark only the important stuff in that split second.
The Magic: The robot is trained to ignore boring, static things (like the wall or the table) and only highlight things that change or matter (like the gripper moving or the cup being lifted).
Analogy: Imagine reading a long novel. Instead of photocopying every single page to remember the story, you just write a one-sentence summary on a sticky note for every chapter. "Chapter 1: Hero enters the cave." "Chapter 2: Hero finds a sword."

B. The Memory Module (The "Diary")

These "sticky notes" (Moment Tokens) are fed into a Memory Module.

How it works: This module is like a smart librarian. It doesn't just stack all the notes in a pile. It looks at the current situation and asks, "Which of my past notes are actually useful right now?"
The Magic: If the robot is currently confused about where a cup is, the Memory Module says, "Hey, look at the note from 5 seconds ago! That's when we saw the blue cup." It pulls that specific memory to help make the next decision.
Analogy: It's like having a conversation with a friend who has a perfect memory. You say, "I think I lost my keys." Your friend (the Memory Module) says, "Wait, I remember you put them on the counter when you walked in." You didn't need to remember everything; your friend just gave you the right piece of information at the right time.

3. Why is this better than just showing the robot more pictures?

You might think, "Why not just show the robot the last 10 photos so it can see the history?"

The Old Way (Multi-frame): This is like trying to read 10 pages of a book at once by squinting. It's heavy, slow, and the robot gets confused by too much visual noise. It slows down the robot's brain and uses up a lot of computer memory.
The HAMLET Way: This is like reading the summary notes. It's lightweight and fast. The robot only carries the "essence" of the past, not the raw data. It's like carrying a pocket-sized diary instead of a library.

4. The Results: From Clumsy to Capable

The researchers tested HAMLET on real robots doing tricky tasks, like:

Cover and Stack: Covering a cube with a cup, then stacking another cup on top.
Swap Cubes: Moving two cubes around in a specific order.

The Outcome:

Without HAMLET, the robot was often confused, dropping items or moving the wrong object.
With HAMLET, the robot became a master chef of chores. In one test, the success rate jumped from 29% to 76%. That's like going from failing a driving test three times out of four to passing almost every time.

The Big Takeaway

HAMLET is a "plug-in" upgrade. You don't need to rebuild the robot's brain or teach it from scratch. You just give it a smart memory system that knows how to summarize the past and recall the right details when needed.

It turns a robot that lives only in the "now" into a robot that understands the "story" of what it's doing, making it much better at complex, long-term tasks.

1. Problem Statement

Vision-Language-Action (VLA) models have shown promise in robotic control by leveraging large-scale pre-trained Vision-Language Models (VLMs). However, most existing VLAs operate under a single-frame assumption, predicting actions solely based on the current observation ( $o_t$ ).

This approach faces critical limitations:

History-Dependence: Robotic manipulation tasks are inherently non-Markovian. Decisions often depend on past context (e.g., whether an object was already grasped, or if it is currently occluded).
Computational Cost of Naive Solutions: Simply appending multiple past frames to the VLA input (Multi-frame approach) incurs massive computational overhead. The paper notes that adding just four past frames increases inference latency by ~35% and peak memory consumption by ~3.6×, severely limiting scalability and batch sizes.
Causal Confusion: Naive multi-frame inputs can lead to spurious temporal correlations, causing the model to fail in dynamic environments.

The core research question is: How can we integrate history-awareness into pre-trained VLAs without costly retraining from scratch or prohibitive computational overhead?

2. Methodology: HAMLET

The authors propose HAMLET (History-Aware Memory with LEarned Tokens), a plug-in fine-tuning framework that transforms pre-trained VLAs into history-aware policies. It consists of two main components:

A. Moment Tokens (Context Compression)

Instead of storing raw past images, HAMLET compresses the perceptual information of each timestep into a compact representation called Moment Tokens ( $m_t$ ).

Mechanism: At each timestep $t$ , a set of learnable tokens is appended to the VLM input alongside the current observation and task instruction.
Initialization via Time-Contrastive Learning (TCL): To ensure these tokens capture temporally distinctive features rather than static background noise, they are initialized using a contrastive learning objective.
- Positive Pairs: Augmented views of the same timestep.
- Negative Pairs: Views from different timesteps within the same trajectory.
- Goal: This forces the tokens to align with representations from the same moment while remaining discriminative across time, effectively filtering out static backgrounds and emphasizing task-relevant dynamics.

B. Lightweight Memory Module

A lightweight Transformer module aggregates these moment tokens across time to create a history-aware condition for action prediction.

Architecture: A shallow Transformer (2 layers) uses causal self-attention to process a stack of recent moment tokens ( $m'_{t-k(T-1)} \dots m'_t$ ).
Selective Attention: Unlike simple concatenation (which the paper shows is suboptimal), this module learns to selectively attend to informative past timesteps while ignoring irrelevant ones.
Integration: The output of the memory module (a history-augmented feature $\tilde{m}'_t$ ) is concatenated with the current VLM hidden state ( $h_t$ ) and fed into the Action Expert (e.g., a Diffusion head) to predict the next action chunk.

3. Key Contributions

Framework Design: Introduced HAMLET, an easily integrable, plug-in framework that enables pre-trained VLAs to leverage historical context without retraining the backbone from scratch.
Novel Token Mechanism: Proposed Moment Tokens initialized with Time-Contrastive Learning to compress temporal information efficiently, suppressing static redundancy.
Efficient Memory Architecture: Designed a lightweight Transformer-based memory module that selectively aggregates historical cues, avoiding the computational explosion of multi-frame inputs.
Empirical Validation: Demonstrated significant performance gains across diverse real-world and simulation benchmarks, proving the method's generalizability across different VLA backbones (e.g., GR00T, CogACT).

4. Experimental Results

Real-World Tasks (History-Dependent)

Evaluated on three long-horizon tasks requiring reasoning over past trajectories (Pick-and-Place Twice, Cover-and-Stack, Swap Cubes) using the GR00T N1.5 backbone.

Performance: HAMLET achieved an average success rate of 76.4%, surpassing the baseline GR00T N1.5 (29.2%) by 47.2%.
Comparison: It significantly outperformed the naive "Multi-frame" baseline (45.8%), which struggled with occlusion and state ambiguity.
Qualitative: Visualizations showed HAMLET correctly identifying occluded objects and recalling previous steps (e.g., "The blue cube was on the right"), whereas the baseline failed to recover from errors.

Simulation Benchmarks

RoboCasa Kitchen (100 demos): HAMLET improved success rates from 64.1% (baseline) to 66.4%.
LIBERO: Pushed the state-of-the-art from 95.6% to 97.6% (near-saturated performance).
SimplerEnv-Bridge (CogACT): Improved average success rate from 52.1% to 63.5%.

Efficiency Analysis

Latency & Memory: HAMLET maintains near-single-frame efficiency.
- At a history length of 8, the Multi-frame baseline increased latency by 2.4× and memory by 7×.
- HAMLET increased latency by only 1.07× and memory by 2.0×.
Transferability: The memory module trained on one dataset (LIBERO) successfully transferred to another (RoboCasa) with minimal performance drop, indicating learned representations are generalizable.

5. Significance

HAMLET addresses a fundamental bottleneck in robotic learning: the trade-off between contextual awareness and computational efficiency.

Scalability: By compressing history into learnable tokens and using a lightweight memory module, it allows state-of-the-art VLAs to handle long-horizon, non-Markovian tasks without the prohibitive cost of processing raw video frames.
Plug-and-Play: The framework does not require pre-training from scratch; it can be applied to existing pre-trained models (both diffusion-based and autoregressive) via fine-tuning.
Practical Impact: The results suggest that enabling "history-awareness" is critical for reliable real-world robot deployment, particularly for tasks involving occlusion, multi-step reasoning, and recovery from failure states.

In conclusion, HAMLET provides a scalable, efficient, and effective pathway to upgrade current Vision-Language-Action models into robust, history-aware policies capable of tackling complex, long-horizon robotic manipulation tasks.