💻 computer science

SkillWrapper: Generative Predicate Invention for Task-level Robot Planning

This paper introduces SkillWrapper, a method that leverages foundation models to automatically invent formal, provably sound and complete symbolic predicates from raw RGB observations, enabling robots to generalize black-box skills into abstract representations for solving unseen long-horizon tasks.

Original authors: Ziyi Yang, Benned Hedegaard, Ahmed Jaafar, Yichen Wei, Skye Thompson, Shreyas S. Raman, Haotian Fu, Stefanie Tellex, George Konidaris, David Paulius, Naman Shah

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ziyi Yang, Benned Hedegaard, Ahmed Jaafar, Yichen Wei, Skye Thompson, Shreyas S. Raman, Haotian Fu, Stefanie Tellex, George Konidaris, David Paulius, Naman Shah

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a robot that is incredibly strong and dexterous at the physical level. It can pick up a teapot, pour tea, and stack a bowl on a plate. However, right now, this robot is like a brilliant pianist who only knows how to press individual keys but doesn't understand music theory. It can play a single note, but it doesn't know how to compose a song or plan a whole concert.

This is the problem SkillWrapper solves.

The Problem: The "Black Box" Robot

Currently, many robots come with "black box" skills. These are pre-programmed actions (like "Pick up object" or "Pour liquid") that work well on their own. But if you want the robot to do a complex, multi-step task—like "Make a cup of tea"—it doesn't know how to string these skills together.

To get a robot to plan a complex task, humans usually have to manually write a rulebook for the robot. They have to explain: "To pour tea, the robot must first be holding the teapot, and the cup must be empty." This is tedious, slow, and impossible to do for every new robot or environment.

The Solution: Teaching the Robot to Write Its Own Rulebook

The researchers created a system called SkillWrapper. Instead of a human writing the rulebook, SkillWrapper teaches the robot to invent its own "vocabulary" and "rules" by watching itself try things out.

Here is how it works, using a simple analogy:

1. The "Trial and Error" Phase (Active Data Collection)

Imagine a robot in a kitchen. It doesn't know the rules yet. SkillWrapper tells the robot: "Go try picking up the teapot. Now try picking up the sponge. Now try stacking the bowl."

Success: The robot picks up the teapot.
Failure: The robot tries to pick up the sponge, but it's too slippery, and it drops it.

The robot records these moments: "I tried to pick up the teapot, and it worked. I tried to pick up the sponge, and it failed."

2. The "Aha!" Moment (Generative Predicate Invention)

This is the magic part. The robot looks at its successes and failures and asks: "Why did one work and the other fail?"

In the past, computers needed humans to tell them the answer (e.g., "The sponge is too heavy"). But SkillWrapper uses a Foundation Model (a super-smart AI that understands images and language, like a very advanced version of ChatGPT or DALL-E).

The robot shows the AI two pictures: one of a successful pickup and one of a failed pickup.
The AI looks at the images and invents a new word (a predicate) to describe the difference.
The AI says: "Ah! The difference is that the teapot has a handle, but the sponge is slippery. Let's invent a new rule called GripperEmpty or ObjectIsStable."

The robot creates a new, human-readable concept that explains why the action worked or failed. It's like the robot suddenly learning the word "slippery" and realizing, "Oh, I can't pick up slippery things with my current grip!"

3. Building the Rulebook (Operator Learning)

Once the robot has invented these new words (predicates), it starts building a logical map.

Rule: "To Pour, you must be Holding a Teapot and the Cup must be Empty."
Rule: "To Stack, the Plate must be Flat."

The robot groups these rules together to form a Symbolic Model. This is a high-level map of the world that ignores the messy details (like the exact angle of the arm) and focuses on the logic (like "is the cup empty?").

4. The Grand Plan (Task-Level Planning)

Now, when you give the robot a complex goal like "Make tea," it doesn't panic. It uses a standard planning algorithm (the same kind of logic used in video game AI or logistics software) to look at its new rulebook.

It sees the goal: "Cup has tea."
It checks the rules: "I need to Pour."
It checks the preconditions: "Do I have a teapot? Is the cup empty?"
It builds a step-by-step plan: Pick up Teapot -> Move to Cup -> Pour.

Why This Paper is Special

The paper claims three major things that make this different from previous attempts:

It starts from zero: Unlike other methods that need humans to give them a starting list of rules, SkillWrapper starts with nothing. It invents the vocabulary from scratch just by watching the robot move.
It's mathematically guaranteed to be correct: The authors didn't just guess that this would work. They proved with math that if the robot follows this process, the rules it learns will be Sound (it won't plan impossible things) and Complete (it won't miss a solution if one exists). It's like proving a bridge is safe to cross before letting anyone walk on it.
It works in the real world: They tested this on real robots (not just simulations). The robots learned to plan complex tasks, like stacking objects or pouring liquids, using only camera images as input.

The Bottom Line

SkillWrapper is a system that lets a robot teach itself the "grammar" of its own actions. Instead of a human writing a manual, the robot tries things, fails, asks a smart AI "Why did that fail?", invents a new word to describe the problem, and then uses that word to plan its next move. This allows robots to solve long, complicated tasks in the real world without needing a human expert to program every single rule.

Technical Summary: SkillWrapper: Generative Predicate Invention for Task-level Robot Planning

Problem Statement

Building autonomous robots capable of long-horizon task planning requires bridging the gap between low-level sensory/motor control and high-level abstract reasoning. While humans can reason about complex tasks (e.g., cooking) without modeling muscle-level control, robots typically rely on manually specified task-level models (preconditions and effects) defined by domain experts. In real-world settings, acquiring these models is nontrivial due to complex inter-skill constraints and specific environmental embodiments.

Existing approaches to learning these abstractions face a dichotomy:

Bottom-up methods: Offer theoretical guarantees (soundness and completeness) but often rely on restricted, structured state spaces (e.g., configuration space) or struggle to scale from pixel-level inputs without foundation models.
Foundation Model (FM) driven methods: Leverage FMs to generate semantically meaningful predicates from raw sensory inputs (a process termed generative predicate invention). However, these methods typically rely on heuristic or ad-hoc procedures, lacking formal guarantees regarding which properties the learned abstractions must satisfy or how to ensure the resulting models are correct for planning.

The core challenge addressed by this paper is: How can we autonomously learn sound and complete symbolic representations of black-box robot skills from raw RGB observations, using foundation models, while providing formal guarantees for task-level planning?

Methodology: SkillWrapper

The authors propose SkillWrapper, an active learning framework that autonomously acquires sound and complete symbolic models for black-box skills. The system operates on a domain defined by a set of object-centric skills ( $\Omega$ ) and learns an abstract transition model $M = \langle P, A \rangle$ , where $P$ is a set of predicates and $A$ is a set of operators.

The method iterates through three primary steps:

1. Active Data Collection

Instead of random exploration, SkillWrapper actively selects sequences of skill instances to execute. It prompts a Foundation Model (FM) to generate candidate skill sequences, which are then scored based on two axes:

Balance ( $b$ ): Prioritizes sequences where the ratio of successful to failed executions is close to a target (e.g., 0.5). This ensures the collection of both positive and negative examples, which is critical for contrastive learning.
Coverage ( $c$ ): Prioritizes sequences that maximize the entropy of consecutive skill pairs observed. This ensures the system observes diverse inter-skill dependencies.
The system selects the Pareto-optimal subset of sequences and executes them, recording transitions $\langle s, \omega, s' \rangle$ that capture executability and outcomes.

2. Generative Predicate Invention

SkillWrapper begins with an empty predicate set ( $P = \emptyset$ ). It invents new predicates only when the current model fails to explain observed transitions. The invention is triggered by two formal conditions:

Condition 1 (Precondition Inconsistency): The model predicts a skill instance will succeed, but it fails (or vice versa), and the current predicates cannot distinguish the successful state from the failed state.
Condition 2 (Effect Inconsistency): The model cannot distinguish between the effects of a successful execution and a failed execution of the same skill (even when accounting for different object arguments via a lifting function).

When triggered, the system prompts an FM (specifically a Vision-Language Model) to generate a candidate predicate that semantically distinguishes the conflicting cases. The candidate is evaluated by checking if it improves the model's ability to explain the dataset (i.e., if it allows for the construction of operators that correctly separate successful and failed transitions).

3. Operator Learning

Using the accumulated data and the current set of predicates, SkillWrapper learns operators (actions with preconditions and effects).

Clustering: Transitions are clustered based on their "lifted abstract changes" (differences in abstract states).
Precondition Derivation: For each cluster, preconditions are derived as the intersection of the abstract initial states of the transitions in that cluster.
Type Handling: To prevent over-generalization in domains with overlapping object categories, the system conservatively assigns operator parameters using the lowest level of the type hierarchy consistent with the data.

Theoretical Framework

The paper establishes a formal theory for generative predicate invention, providing the first provable guarantees for this class of methods:

Soundness: The learned model correctly predicts the effects of a plan; if a planner predicts a sequence reaches a state, the execution truly leads there.
Completeness: The model never omits real solutions; if a solution exists in the environment, the planner can find an abstract plan.
Correctness: The authors prove that SkillWrapper's learned model is empirically correct with respect to the observed data (Theorem 1) and asymptotically converges to a correct model under a stationary $\beta$ -mixing process (Theorem 2).

Experimental Results

The authors evaluated SkillWrapper in both simulation and on real robots.

Simulation (Robotouille)

In a simulated kitchen environment with six objects and five skills, SkillWrapper was compared against:

Expert Operators: Manually defined PDDL models.
System Predicates: Ground-truth simulator predicates (oracle).
ViLa: A baseline using direct FM prompting for action selection.
Ablations: Variants without active data collection heuristics or without the specific predicate invention algorithm.

Findings:

SkillWrapper outperformed all non-privileged baselines (including ViLa and random exploration).
On "hard" problems (up to 15 steps), SkillWrapper surpassed the performance of the System Predicates baseline, demonstrating that learned abstractions can be more effective than ground-truth simulator predicates in complex planning scenarios.
It correctly identified 100% of impossible tasks, a requirement for soundness.

Real-Robot Experiments

Experiments were conducted on a Franka Emika Panda (tabletop) and a bimanual Kuka iiwa manipulator.

Generalization: A model learned in restricted training environments (subsets of objects) successfully generalized to a test environment containing all objects, solving unseen long-horizon tasks.
Data Efficiency: In a bimanual setting with dead ends, SkillWrapper improved performance over iterations, surpassing all baselines within five iterations. The system demonstrated that guided active data collection is crucial for learning in environments with irreversible actions.

Significance and Claims

The paper claims to be the first work on generative predicate invention that provably guarantees soundness and completeness for task-level planning.

Key contributions include:

Formal Theory: A characterization of the conditions under which a learned model is provably sound and complete, addressing the lack of theoretical rigor in prior FM-driven approaches.
SkillWrapper System: A principled system that leverages foundation models to interactively collect data, invent predicates, and learn interpretable symbolic representations without manual specification.
Empirical Validation: Extensive evidence showing that these learned representations enable robots to compose black-box skills to solve unseen, long-horizon tasks in the real world using off-the-shelf classical planners.

The authors conclude that SkillWrapper offers a promising path toward scalable symbolic model learning, enabling robots to reason about high-level objectives without requiring hand-engineered abstractions or privileged knowledge of the environment's dynamics. The work acknowledges limitations, such as the assumption of deterministic dynamics and the inability to directly control FM outputs, suggesting future work in probabilistic settings and noise handling.