🤖 machine learning

BooST: Bridging Semantics and Motions for Efficient Skill Transfer

BooST is a two-stage framework that bridges high-level semantic intent and low-level motion dynamics via a cross-modal VQ-VAE to create a unified skill representation, enabling efficient few-shot adaptation, cross-domain transfer, and robustness for real-world robot deployment.

Original authors: Jusuk Lee, Daesol Cho, Jonghun Shin, Seungyeon Yoo, Jonghae Park, Taekbeom Lee, H. Jin Kim

Published 2026-08-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jusuk Lee, Daesol Cho, Jonghun Shin, Seungyeon Yoo, Jonghae Park, Taekbeom Lee, H. Jin Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to do chores. You could try to show it every single tiny movement for every single task—like teaching a child to tie a shoe by describing the exact angle of every finger for every second. That takes forever and doesn't work well if the shoes are a different color or the room is messy. This is the big puzzle in robot learning: how do we teach machines to be smart and flexible without needing a million hours of practice?

Scientists have been trying to solve this by teaching robots "skills." Think of a skill like a pre-made recipe or a dance move. Instead of learning how to walk step-by-step from scratch, a robot learns a "walking skill" it can reuse. But there's a catch. Some methods teach the robot how to move (the muscle memory), but they don't understand what it's supposed to do (the goal). Other methods understand the goal perfectly but don't know how to actually move the body to get there. It's like having a chef who knows exactly what a cake should taste like but has no idea how to mix the ingredients, or a baker who can mix anything but doesn't know what a cake is supposed to look like. To make robots truly useful in our messy, unpredictable real world, we need a way to teach them both the "what" and the "how" at the same time, so they can learn new tasks quickly and ignore distractions like a cat jumping on the counter.

This is where a new framework called BooST (Bridging Semantics and Motions for Efficient Skill Transfer) comes in. The researchers behind BooST realized that to make robots efficient, they needed to stop treating "what to do" and "how to do it" as separate problems. They built a two-stage training system that acts like a master chef teaching an apprentice.

In the first stage, called Unified Skill Pretraining, BooST learns from a massive library of robot videos (76,000 episodes, to be exact). It uses a special "translator" that looks at two things at once: the visual scene and language instructions (like "pick up the apple") to understand the intent, and the actual robot movements to understand the dynamics. Imagine a robot watching a video of someone picking up a cup. BooST doesn't just see the hand moving; it understands that the goal is to lift the cup, and it learns to ignore if there's a dog running in the background or if the lighting changes. It compresses all this information into a compact "skill code"—a digital shorthand that captures the essence of the action.

In the second stage, Downstream Adaptation, this heavy, smart knowledge is distilled into a lightweight, fast policy. This is like taking a master chef's years of experience and turning it into a simple, easy-to-follow recipe card that a beginner can use immediately. When the robot faces a new task with very few examples (as few as 5 demonstrations in real-world tests), it doesn't need to relearn everything. It just looks at its "skill codebook," picks the right move, and adapts quickly.

The paper shows that this approach works incredibly well. In computer simulations, BooST outperformed other methods, especially when data was scarce. For instance, when given only 10 demonstrations of a task, BooST was 140% better than the second-best method at adapting to new scenarios. Even more impressively, the researchers tested this on a real robot arm (a UR3) that was different from the one used to gather the training data (a Franka Emika Panda). Despite the difference in body type and movement style, BooST successfully transferred the skills, achieving high success rates with just 5 demonstrations per task.

The researchers also tested how well the robot handled chaos. They added moving, distracting objects (like a human figure walking around) to the training videos. While other methods got confused by the noise and failed to learn the right moves, BooST remained focused on the task, proving it could ignore irrelevant visual distractions. The study suggests that by bridging the gap between understanding the goal and mastering the movement, robots can become much more adaptable, robust, and ready for the real world, all while using less computing power and data than before.

Technical Summary: BooST – Bridging Semantics and Motions for Efficient Skill Transfer

Problem Statement

The paper addresses the challenge of efficient skill transfer in robot learning. While skill abstraction (learning reusable, temporally extended behaviors) offers a promising path toward general-purpose agents, existing methods struggle to simultaneously satisfy three critical desiderata for real-world deployment: generalization, robustness, and efficiency.

Current approaches typically capture only one aspect of a skill:

Low-level methods focus on motion dynamics (how) but lack semantic grounding, hindering transfer across different robotic embodiments and tasks.
High-level methods focus on semantic intent (what) derived from vision and language but often lack motion fidelity, making them sensitive to visual distractors and requiring large-scale models that are computationally prohibitive for real-time deployment.

This incompleteness results in weak priors for policy learning, necessitating substantial in-domain data for adaptation. The authors argue that a unified representation capturing both semantics and motions is required to enable few-shot adaptation, cross-domain transfer, and robustness to dynamic visual noise.

Methodology: BooST Framework

The authors propose BooST (Bridging Semantics and Motions for Efficient Skill Transfer), a two-stage framework designed to learn a unified skill representation and distill it for efficient downstream use.

Stage I: Unified Skill Pretraining

The first stage involves pretraining a cross-modal VQ-VAE on large-scale, diverse offline datasets (specifically the DROID dataset). The goal is to learn a discrete latent space of reusable skills that jointly encodes high-level semantics and low-level motion dynamics. The architecture employs two complementary pathways:

Visuo-linguistic Pathway: Uses a pretrained CLIP model and a cross-attention mechanism to extract task-aware visual features from current and future images ( $I_t, I_{t+H}$ ) conditioned on language instructions ( $l$ ). This pathway grounds the skill in semantic intent.
Action Pathway: Encodes motion dynamics directly from action trajectories ( $a_{t:t+H}$ ) to ensure the learned skills are executable and grounded in physical behavior.

These pathways feed into a shared discrete codebook via Vector Quantization (VQ). The model is trained using a residual vector quantization scheme with a single supervisory signal: action reconstruction. The decoder reconstructs action sequences from the discrete skill token and the current visual features. The loss function combines reconstruction errors from both pathways and a regularization term to align the feature spaces. Crucially, by reconstructing actions rather than pixels, the model is encouraged to ignore task-irrelevant visual details, enhancing robustness to dynamic distractors.

Stage II: Downstream Adaptation

The second stage focuses on distillation for efficient deployment. The rich knowledge from the pretrained skill encoder ( $q_\phi$ ) is transferred to two lightweight components:

A Causal Skill Prior ( $p_\psi$ ) that predicts the distribution of latent skills given past observations and language.
A Low-level Policy ( $\pi_\theta$ ) that executes actions conditioned on sampled skills.

This stage is trained on small-scale, in-domain datasets using a variational objective derived from the Evidence Lower Bound (ELBO). The training minimizes a Prior Distillation Loss (aligning the prior with the frozen encoder's predictions) and a Behavior Cloning Loss (training the policy to imitate expert actions). This decoupled approach allows the system to leverage large-scale pretraining while maintaining a lightweight architecture suitable for real-time control.

Key Contributions

Unified Skill Representation: The introduction of BooST, which integrates semantic intent and motion dynamics into a shared skill codebook, overcoming the limitations of methods that capture only one aspect.
Decoupled Two-Stage Pipeline: A training paradigm that reconciles large-scale pretraining with deployment efficiency. It separates the learning of a rich, unified representation from the training of lightweight, task-specific policies, enabling few-shot adaptation.
Robustness and Generalization: Demonstrated ability to achieve strong few-shot adaptation, cross-embodiment transfer (e.g., from Franka Panda to UR3), and robustness to dynamic visual distractors and background variations.

Experimental Results

The authors evaluated BooST in both simulation (LIBERO benchmarks) and real-world settings (UR3 robot).

Few-Shot Adaptation: On the LIBERO benchmarks, BooST consistently outperformed baselines (including Diffusion Policy, VQ-BeT, QueST, LISA, and EXTRACT). Notably, with only 10 demonstrations per task, BooST achieved a 140% relative improvement over the second-best method on the diverse LIBERO-90 benchmark.
Cross-Embodiment Transfer: In real-world experiments, a policy pretrained on a Franka Emika Panda successfully transferred to a UR3 robot with only five demonstrations per task. Low-level baselines failed to transfer due to action space mismatches, while BooST's semantic grounding enabled successful adaptation.
Robustness to Distractors: When pretrained on data augmented with dynamic human distractors, BooST maintained high success rates on standard benchmarks. In contrast, latent action baselines (LAPA, UniVLA) that reconstruct visual transitions suffered performance drops, as their representations encoded irrelevant background motion.
Efficiency: The distilled downstream models run at approximately 60 Hz, significantly faster than comparable baselines (e.g., Diffusion Policy at 12 Hz), making them suitable for real-world deployment. Ablation studies confirmed that both the action pathway and the task-aware encoder are critical for performance.

Significance

The paper claims that BooST successfully satisfies the three key desiderata for efficient skill transfer—generalization, robustness, and efficiency—which existing methods fail to achieve simultaneously. By bridging the gap between "what" (semantics) and "how" (motions), BooST provides a strong prior that drastically reduces the data required for downstream adaptation. The authors posit that this approach enables the transfer of skills learned from large-scale, heterogeneous datasets to novel scenes, tasks, and robotic embodiments with minimal in-domain data, while remaining robust to the visual noise inherent in real-world environments. The work suggests a path toward practical, general-purpose robotic agents that can learn from "in-the-wild" data without requiring manual curation or computationally expensive models for real-time control.