💻 computer science

ICI-VLA: In-Context Imitation with Spatiotemporally Aligned Demonstrations for Vision-Language-Action Models

ICI-VLA is a training and retrieval framework that enables fixed Vision-Language-Action models to achieve rapid, few-shot test-time adaptation by conditioning action generation on spatiotemporally aligned, semantically labeled micro-demonstrations, thereby eliminating the need for additional gradient updates while significantly outperforming baselines on multiple robotic manipulation benchmarks.

Original authors: Songhua Yang (Wuhan University), Ziyu Liu (Wuhan University), Xuetao Li (Wuhan University), Ruqi Xiao (Wuhan University), Kangxin Zhu (Wuhan University), Miao Li (Institute of Technological Sciences

Published 2026-09-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Songhua Yang (Wuhan University), Ziyu Liu (Wuhan University), Xuetao Li (Wuhan University), Ruqi Xiao (Wuhan University), Kangxin Zhu (Wuhan University), Miao Li (Institute of Technological Sciences, Wuhan University)

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots have long struggled to learn new tasks quickly. Traditional methods often require a robot to be retrained from scratch every time it faces a new job, a process that demands vast amounts of data and computing power. This makes it difficult to deploy robots in changing environments where they must adapt on the fly. A newer approach, known as in-context learning, offers a different path. Instead of retraining the robot's brain, this method allows the system to look at a few examples of how a task was done before and use that information to figure out what to do next, all without changing its internal settings. While this technique has revolutionized how computers understand language and images, applying it to physical robots is far more complex. A robot must not only understand a visual scene and a spoken instruction but also coordinate its own body movements in real time, where a slight mismatch in timing or position can cause a failure.

Researchers at Wuhan University have developed a new framework called ICI-VLA that successfully brings this "learn from examples" capability to robotic arms. Their system allows a robot to watch a few short video clips of a task being performed and then immediately apply that knowledge to a new, similar situation. The key innovation lies in how the system handles the examples. Instead of feeding the robot entire, long videos of a task, the researchers break these demonstrations down into tiny, labeled segments that match specific moments in the robot's current job. They then train a specialized search tool to find the exact segment that aligns with what the robot is seeing right now, ensuring the robot copies the right motion at the right time. To prevent the robot from simply memorizing the numbers in the example videos, the system is trained to ignore the exact end of the example and instead focus on the current view, forcing it to understand the action rather than just repeat it.

In their experiments, the team tested this approach in two different simulated environments and on a real physical robot with two arms. In the first simulation, which involved a variety of tasks like moving objects and opening drawers, the system achieved a success rate of 97.7 percent. In a second, more difficult simulation involving dual-arm coordination, it reached 60.4 percent, a significant improvement over previous methods. When they moved the system to a real-world setup with physical cameras and robotic arms, it successfully completed tasks like sorting objects and placing items into drawers 83.2 percent of the time. These results suggest that a robot does not need to be retrained for every new job if it can be given the right, well-matched examples to look at in the moment.

The core of this success is how the system manages the flow of information. When a robot is given a long instruction, such as "open the drawer and put the bowl inside," the system breaks this down into smaller steps. It then searches a library of past experiences to find short clips that match the current step. For instance, if the robot is currently trying to push a drawer closed, the system finds a short clip of a drawer being pushed closed, rather than showing it a clip of the drawer being opened. This ensures the robot is looking at relevant information. The researchers also introduced a training technique where they hide parts of the example actions during the learning phase. This forces the robot to rely on what it is seeing right now and the general context of the task, rather than just blindly copying the numbers from the example video. This prevents the robot from getting confused if the starting position is slightly different from the example.

The study highlights that the quality of the examples matters more than the quantity. By carefully selecting and aligning these short demonstrations with the robot's current phase of movement, the system can adapt instantly. The researchers found that using just three examples was enough to achieve peak performance; adding more did not help and sometimes made the system less effective. This indicates that the robot benefits from a few, highly relevant clues rather than a flood of information. The system works by keeping the robot's main control software fixed and unchanged, only adjusting its behavior based on the retrieved examples. This means the robot can be deployed in new situations without the need for expensive and time-consuming retraining sessions.

While the results are promising, the researchers note that the system still depends on having a good library of past demonstrations to draw from. If the robot encounters a situation that is completely unlike anything in its library, it may struggle to find a useful example. Additionally, the process of building the library and training the search tool requires significant offline work before the robot can be used. However, the findings demonstrate a clear path forward for making robots more flexible. By treating past experiences as a reference guide rather than a rigid script, robots can learn to handle new challenges with a level of adaptability that was previously out of reach. The work suggests that the future of robotic control may lie not in building smarter brains from scratch, but in teaching them how to learn from the right examples at the right time.

Technical Summary: ICI-VLA

Problem Statement
Vision-Language-Action (VLA) models typically adapt to new manipulation tasks by collecting task-specific data and performing gradient updates (fine-tuning). This paradigm limits rapid deployment in dynamic environments where data or compute is scarce. While In-Context Learning (ICL) offers a parameter-free alternative by conditioning on input demonstrations, applying it to robotics is challenging. Robotic demonstrations involve coupled visual, linguistic, proprioceptive, and continuous action data. Poorly matched context can lead to "temporal aliasing" (misalignment between long trajectories and short action chunks) or "semantic-dynamic mismatch" (visually similar demonstrations requiring incompatible motions), causing the model to copy sequences rather than reason from current observations. Furthermore, existing VLA architectures often rely on specialized action heads or tokenizers that increase engineering overhead and may degrade the foundation model's native reasoning capabilities.

Methodology
The authors propose ICI-VLA (In-Context Imitation with Spatiotemporally Aligned Demonstrations), a framework that enables few-shot test-time adaptation using a fixed text-action policy. The approach consists of three main stages:

Micro-Demonstration Construction: Long-horizon trajectories are decomposed into short, semantically labeled "micro-demonstrations." A Qwen3-VL model heuristically segments trajectories into subtasks, pairing each with fine-grained instructions, camera observations, proprioceptive states, and textualized action chunks. This reduces temporal mismatch between the query and the retrieved context.
Spatiotemporal Retrieval Alignment (RD-Encoder): To retrieve relevant context, the framework trains a Retrieval-Demonstration (RD) Encoder using a two-step mining process:
- Semantic Hard-Filtering: Filters candidates based on semantic compatibility with the current subtask instruction.
- DTW Geometric Ranking: Uses Dynamic Time Warping (DTW) on labeled trajectories to rank the semantically filtered pool. This ensures retrieved examples match both the task semantics and the required motion phase/geometry.
- The encoder is trained via contrastive learning using positives mined by DTW and hard negatives (phase-misaligned candidates from the same semantic subtask).
Policy Training with Target Action Masking: The VLA policy (based on Qwen3-VL-4B) is fine-tuned without adding any action-specific heads or structural modifications, preserving the native text-generation interface. A key innovation is Target Action Masking, a context-corruption objective where target action tokens in the retrieved demonstrations are masked during training. This forces the model to rely on the current observation and the semantic context of the demonstration rather than simply copying the numerical continuation of the action sequence.

At inference, the policy and retriever remain frozen. The system conditions action generation on the current observation and a small set of retrieved micro-demonstrations, requiring no gradient updates.

Key Contributions

Fixed Policy Adaptation: Formulates VLA adaptation as few-shot test-time conditioning on a fixed text-action policy, explicitly separating offline training from test-time inference.
Subtask-Level Decomposition & Retrieval: Constructs subtask-level micro-demonstrations and employs DTW-mined supervision to train an RD-Encoder capable of semantic and phase-aware retrieval.
Target Action Masking: Introduces a context-corruption objective to reduce direct action copying and increase reliance on current observations, validated through ablation studies.

Experimental Results
ICI-VLA was evaluated on LIBERO, RoboTwin 2.0, and four physical dual-arm Aloha tasks:

LIBERO: Achieved an average success rate of 97.7%, outperforming the highest reported baseline average (96.4% from OpenVLA-OFT) in the benchmark-level comparison.
RoboTwin 2.0: Achieved 60.4% average success, exceeding the strongest reported baseline average by 19.3 percentage points. Under controlled conditions (same backbone and data), it significantly outperformed a basic ICL variant (10.7%).
Physical Tasks: Achieved an average success rate of 83.2% across four physical dual-arm tasks, surpassing baselines like $\pi0$ (66.4%) and VLA-0 (63.0%).
Ablations: Removing Target Action Masking caused a drastic drop in performance (e.g., from 60.4% to 10.7% on RoboTwin 2.0), confirming its necessity for preventing trajectory copying. Similarly, removing DTW or semantic filtering significantly degraded retrieval quality and success rates.

Significance and Claims
The paper claims that ICI-VLA demonstrates that a fixed VLA policy can effectively adapt to new tasks at test time by conditioning on spatiotemporally aligned demonstrations, eliminating the need for task-specific fine-tuning. By retaining the native text-generation interface and using a retrieval mechanism that aligns both semantics and motion phases, the framework bridges the gap between large-scale pretraining and specific robotic control. The results suggest that "demonstrations as context" is a viable alternative to "demonstrations as training data" for rapid robotic adaptation, provided the context is carefully aligned and the policy is trained to resist simple sequence copying.

Limitations
The authors acknowledge that ICI-VLA remains dependent on the coverage of the demonstration library and the visual planner. It incurs offline training and retrieval costs. Future work is identified as improving retrieval efficiency, expanding demonstration coverage, and investigating scalable cross-embodiment adaptation.

Technical Summary: ICI-VLA

More like this