🤖 machine learning

InSight: Self-Guided Skill Acquisition via Steerable VLAs

InSight is a framework that enables Vision-Language-Action (VLA) models to autonomously acquire new manipulation skills by decomposing demonstrations into steerable primitive actions and utilizing a VLM-guided data flywheel to generate, label, and integrate missing primitives for novel, long-horizon tasks without further human intervention.

Original authors: Maggie Wang, Lars Osterberg, Stephen Tian, Ola Shorinwa, Jiajun Wu, Mac Schwager

Published 2026-06-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Maggie Wang, Lars Osterberg, Stephen Tian, Ola Shorinwa, Jiajun Wu, Mac Schwager

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to cook. Traditionally, if you want the robot to learn a new dish, like "making a sandwich," you have to stand there, hold its hand, and guide it through every single step: pick up the bread, spread the butter, add the cheese, and so on. This is slow, expensive, and if you want it to learn "making soup" later, you have to do the whole process all over again.

The paper INSIGHT proposes a smarter way to teach robots, one that works more like how humans learn new skills. Instead of memorizing whole recipes, the robot learns individual moves (like "pick up," "lift," "turn," or "pour") and then figures out how to combine them on its own.

Here is how the system works, broken down into simple steps:

1. The "Lego Brick" Approach

Think of a robot's skills not as one giant, unbreakable block, but as a box of Lego bricks.

The Problem: Most robots today are trained on whole tasks. If you show them how to "pick up a cup," they learn that specific sequence. If you ask them to "pour water," they might fail because they've never seen the "pouring" brick before.
The INSIGHT Solution: The system takes human demonstrations (like a video of someone picking up a cup) and automatically chops them up into tiny, labeled "bricks" called primitives.
- Example: Instead of seeing "pick up cup," the robot sees: "Move hand to cup" + "Close fingers" + "Lift up."
- The robot learns these individual moves so well that it can be "steered" to do just one of them on command.

2. The "Smart Assistant" (The VLM)

The robot has a built-in "Smart Assistant" (a Vision-Language Model, or VLM) that acts like a project manager.

The Scenario: Imagine the robot knows how to "pick up a bottle" and "put it down," but you ask it to "pour the bottle into a bowl."
The Gap: The Smart Assistant looks at the plan and says, "Hey, we have the 'pick up' brick and the 'put down' brick, but we are missing the 'tilt and pour' brick!"
The Fix: Instead of asking a human to show it how to pour, the Smart Assistant figures out the physics of the move (e.g., "Tilt the bottle forward 45 degrees") and tells the robot to try it.

3. The "Practice Loop"

This is where the magic happens. The robot tries to perform that missing move on its own.

Trial and Error: The robot attempts the "tilt" move. If it spills everywhere, it tries again with a slightly different angle.
The "Oracle" Check: After the robot tries, the Smart Assistant looks at the result (like a teacher grading a test). "Did the water go into the bowl? Yes? Great! No? Try again."
Learning: Once the robot successfully pours the water, the system saves that specific "tilt" move as a new Lego brick in its library. It then re-trains itself to remember this new brick forever.

4. Building New Skills from Scratch

Once the robot has learned the "pour" brick, it doesn't need a human to teach it again.

The Result: If you later ask the robot to "twist a cap off and then pour," it can now combine the "twist" brick (which it might have learned earlier) with the new "pour" brick.
The Analogy: It's like a musician who has learned to play a C-major chord and an F-major chord. If they want to play a new song, they don't need a teacher to show them the whole song; they just combine the chords they already know.

What Did They Actually Prove?

The researchers tested this in both computer simulations and on real robots (using a robotic arm). They showed that:

No New Human Help Needed: They taught the robot to do complex tasks like flipping a block, closing a drawer, twisting a bottle cap, and pouring beans without ever showing it a video of someone doing those specific things.
It Works Fast: The robot learned these new skills much faster than traditional methods (like Reinforcement Learning, which is like a robot trying to learn by blind trial and error thousands of times).
It Doesn't Forget: When the robot learned the new "pouring" skill, it didn't forget how to "pick up" the bottle. It kept all its old skills while adding the new ones.
Real-World Success: On a real robot, it achieved high success rates (up to 96% for pouring) and could even combine skills to do a long, 14-step task (twist a cap, then pour) that it had never seen before.

The Bottom Line

INSIGHT is a framework that lets robots break down complex tasks into small, reusable moves. When they encounter a new job, they use a "Smart Assistant" to figure out which moves they are missing, practice those moves on their own, and save them for later. This allows robots to learn new skills continuously, without needing a human to hold their hand every single time.

Technical Summary: INSIGHT – Self-Guided Skill Acquisition via Steerable VLAs

Problem Statement

Vision-Language-Action (VLA) models have advanced general-purpose robot manipulation but remain fundamentally bounded by the skills present in their training data. Acquiring new manipulation skills typically requires expensive human demonstrations and fine-tuning for every novel task. While reinforcement learning (RL) offers an alternative, it often demands prohibitive sample complexity and safety constraints in the real world. Furthermore, existing methods that utilize Vision-Language Models (VLMs) or Large Language Models (LLMs) as planners or coding agents generally operate at test time; they compose pre-defined skills or generate trajectories without updating the underlying learned policy. Consequently, robots cannot autonomously expand their capability set to handle novel scenarios (e.g., a Mars rover needing to sweep debris when trained only to scoop rocks) without human intervention.

Methodology: INSIGHT Framework

INSIGHT addresses these limitations by rendering VLAs "steerable" at the primitive-action level, enabling autonomous skill acquisition through a two-stage pipeline.

Stage 1: Automatic Primitive Segmentation and Steerable VLA Training

The first stage transforms standard teleoperated demonstrations into a steerable VLA policy without manual annotation.

Primitive Definition: A primitive is defined as a reusable action segment characterized by a single dominant motion mode (e.g., translation along an axis, rotation, or gripper state transition) and a natural language label (e.g., "lift upward," "close gripper").
Automated Segmentation: A VLM is provided with a task description and example primitives to generate an ordered sequence of expected primitives. These VLM-generated plans are aligned with human demonstration data using gripper state transitions (open/close), end-effector motion magnitudes, and dominant-axis tags.
Fine-Tuning: The segmented, labeled trajectories are used to fine-tune a pretrained VLA (using LoRA). Each primitive segment becomes a separate training episode conditioned on its language label. A learned progress channel is added to the action space to provide termination signals for individual primitives.
Outcome: The result is a VLA that can be steered via language to execute specific primitives, rather than just full-task instructions.

Stage 2: VLM-Guided Skill Acquisition Loop

Given a steerable VLA with a known vocabulary of primitives ( $V$ ), INSIGHT autonomously acquires missing skills for novel tasks.

Primitive Gap Identification: For a novel task, a VLM planner decomposes the task into a sequence of primitives. Any primitive $p_i$ in the plan that is not in the current vocabulary $V$ is flagged as a "primitive gap."
Parameterization and Execution: For each gap, the VLM proposes low-level control parameters, specifically a motion axis (translation $\{dx, dy, dz\}$ or rotation $\{dr_x, dr_y, dr_z\}$ ) and a signed magnitude. The robot executes the plan: known primitives are handled by the steerable VLA, while gaps are executed by a low-level controller using the VLM-proposed parameters.
Success Evaluation and Integration: A VLM oracle analyzes the scene before and after the rollout to verify task success. Successful rollouts of the new primitive are automatically labeled, stored, and added to the training dataset.
Retraining: The VLA is retrained on the combined dataset of original and newly acquired primitives. This expands the vocabulary $V$ , allowing the robot to execute the full composed task autonomously in future iterations.

Key Contributions

Automatic Primitive Segmentation Pipeline: A method to decompose teleoperated demonstrations into labeled primitives using VLM plan decomposition and kinematic cues, eliminating the need for manual annotation and enabling primitive-level steerability.
VLM-Guided Data Flywheel: A closed-loop system where the VLM identifies missing primitives for novel tasks, autonomously generates successful demonstrations via low-level control, and retrains the VLA to integrate these new capabilities without further human demonstrations.
Compositional Skill Acquisition: A demonstration that acquired primitives can be composed to solve complex, long-horizon tasks, effectively enabling continual, lifelong learning.

Experimental Results

The framework was evaluated across five tasks in simulation (LIBERO environment with a Franka Panda) and on real-world hardware (UFactory xArm).

Simulation (Block Flipping): Starting from only pick-and-place demonstrations, INSIGHT acquired a "rotate-block" primitive. It achieved 75% success on the block-flip task after 246 acquired primitive rollouts. In contrast, a Soft Actor-Critic (SAC) RL baseline with the same compute budget achieved 0% success on the flip.
Simulation (Drawer Closing): Trained only on drawer-opening demos, INSIGHT acquired a "push drawer closed" primitive from an Out-of-Distribution (OOD) initial state (drawer already open). It achieved 100% success on closing the drawer while retaining the ability to open it.
Real-World (Twisting and Pouring):
- Twisting: INSIGHT achieved 92% end-to-end success on opening a bottle cap, compared to 32% for a Code-as-Policies (CaP-X) baseline and 0% for a standard fine-tuned VLA ( $\pi0.5$ ).
- Pouring: INSIGHT achieved 96% success on pouring beans, versus 16% for CaP-X.
- Composition: For a complex 14-primitive "twist-then-pour" task, INSIGHT achieved 80% end-to-end success, significantly outperforming CaP-X (4%).
- Efficiency: INSIGHT required substantially less wall-clock time per trial than CaP-X (e.g., 13.6s vs 41.5s for twisting) because it executes learned primitives rather than reasoning through every step at test time.
Real-World (Sweeping): From only scooping demonstrations, INSIGHT acquired a lateral-push primitive to sweep debris, succeeding in 5/5 evaluation trials.
Skill Retention: In all experiments, the unified VLA retained 100% performance on its original base skills after acquiring new primitives.

Significance and Claims

The paper claims that primitive steerability provides a practical foundation for continual skill acquisition in VLA policies. By decoupling skills into composable, steerable primitives, INSIGHT allows robots to:

Identify exactly what capabilities are missing for a new task.
Autonomously practice and acquire those specific missing skills.
Integrate new skills into a unified policy without catastrophic forgetting of prior skills.

The authors position this as a shift from treating VLMs merely as test-time planners to using them as active agents in a data acquisition loop that persistently expands the robot's skill set. This approach offers a path toward robots that can adapt to novel environments (like the Mars rover example) without the prohibitive cost of collecting new human demonstrations for every new behavior.

Limitations noted by the authors:

Primitive acquisition is currently restricted to single-axis motions.
The system filters for successful rollouts; incorporating failure analysis could improve sample efficiency.
Human intervention is still required for environment resets between rollouts.
The method has not yet been tested on high-DOF embodiments like humanoids or mobile manipulators.