💻 computer science

Jointly Learning Predicates and Actions Enables Zero-Shot Skill Composition

This paper introduces Predicate Action Skills (PACTS), a closed-loop visuomotor policy that jointly models action trajectories and symbolic predicate outcomes to enable robust zero-shot composition of learned skills through planning.

Original authors: Benedict Quartey, Sebastian Castro, Eric Rosen, Wil Thomason, George Konidaris, Stefanie Tellex

Published 2026-05-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Benedict Quartey, Sebastian Castro, Eric Rosen, Wil Thomason, George Konidaris, Stefanie Tellex

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to cook a complex meal, like making a sandwich and then cleaning up.

The Old Way: The "Blind" Chef
Traditionally, when we teach robots new skills (like "pick up the bread"), we use a method called "Learning from Demonstration." We show the robot a video of a human doing it, and the robot learns to copy the hand movements.

However, there's a problem with this approach. The robot learns the movements perfectly, but it doesn't really understand the result. It knows how to move its arm to grab the bread, but it doesn't have a clear internal check to say, "Okay, I have successfully grabbed the bread."

Because of this, if you ask the robot to do a new combination of skills it hasn't seen before (like "grab the bread, but only if the butter is already open"), it gets confused. It's like a chef who knows how to chop onions but doesn't know what a "chopped onion" looks like, so they can't decide when to stop chopping or what to do next. To fix this, we usually have to retrain the robot from scratch for every new combination.

The New Way: PACTS (The "Mindful" Chef)
This paper introduces a new system called PACTS (Predicate-Action Skills). Think of PACTS as teaching the robot to be a "mindful" chef who learns two things at the exact same time:

The Action: The physical movements (how to move the arm).
The Outcome: A symbolic "belief" about what just happened (e.g., "The bread is now in my hand" or "The door is now open").

The Creative Analogy: The "Dual-Track" Tape
Imagine the robot's learning process is like recording a movie on a special dual-track tape:

Track A records the video of the robot's hands moving.
Track B records a running commentary of "truths" that become true as the movie plays (e.g., "Object is held," "Object is released," "Door is open").

In old systems, these two tracks were recorded by different people who never talked to each other. Sometimes the video showed the robot dropping the cup, but the commentary track still said, "Cup is held." This mismatch caused confusion when trying to plan complex tasks.

With PACTS, the robot records both tracks simultaneously in a single, unified process. As it learns the movement, it also learns the corresponding change in the world. Because they are learned together, they are perfectly synchronized. If the robot's hand moves to pick up a cup, its internal "belief" automatically updates to "I am holding the cup."

Why This Matters: Zero-Shot Composition
The biggest superpower of PACTS is Zero-Shot Composition.

Because the robot now has this clear, symbolic "commentary track" (the predicate beliefs) that updates in real-time, we can use a standard, off-the-shelf "planner" (like a GPS or a recipe book) to mix and match skills the robot has never seen combined before.

The Scenario: You taught the robot to "Open the door" and separately to "Push the cart." You never showed it "Open the door then push the cart."
The Result: The planner looks at the robot's internal "belief track." It sees the robot successfully "Opened the door" (the belief changed from False to True). The planner then says, "Great! Now that the door is open, let's execute the 'Push the cart' skill."

The robot can do this without any new training. It's like a chef who knows how to boil water and how to fry an egg. Even if they've never made a "boiled-egg-fried-sandwich" before, they can look at their internal checklist, see the water is boiled, and immediately switch to frying the egg.

Real-World Testing
The authors tested this in three ways:

A 2D Game: A robot pushing blocks. PACTS was better at both moving the blocks correctly and correctly identifying when the blocks were in the right spot, compared to robots that only learned movements.
3D Simulation: Complex tasks like making coffee or cooking in a kitchen. PACTS maintained high performance while giving the planner a clear view of what was happening.
Real Life: A real robot packing colored cubes into a box. The team taught it to pack red cubes and green cubes separately. Using PACTS, they could then ask the robot to "Pack only the red and yellow cubes" (a combination it had never seen), and it successfully did so by following the planner's instructions based on its internal beliefs.

In Summary
PACTS teaches robots to learn actions and outcomes together, like a single package. This gives the robot a clear, real-time "status report" of what it has achieved. This status report allows a simple planner to mix and match skills instantly, letting the robot solve new, complex problems without needing to be retrained from scratch.

Technical Summary: Jointly Learning Predicates and Actions Enables Zero-Shot Skill Composition

1. Problem Definition

Learning from Demonstration (LfD) has enabled robots to acquire complex behaviors, yet existing generative visuomotor policies often fail to generalize to new compositions of known skills without retraining. Current approaches typically model distributions over action trajectories alone. While effective for capturing multi-modal behavior, these models lack the ability to reason about the symbolic outcomes (state changes) induced by actions. This limitation creates a gap in long-horizon tasks where distribution shift compounds, and the space of task variations grows combinatorially.

Prior work attempting to bridge this gap often decouples components: training one model to generate actions and a separate model to classify predicates from observations. This separation ignores the tight coupling between manipulation actions and their symbolic effects. Consequently, inference-time inconsistencies can arise where the predicted symbolic state does not align with the actual action mode, hindering robust planning-based composition.

2. Methodology: Predicate-Action Skills (PACTS)

The authors propose Predicate-Action Skills (PACTS), a class of closed-loop visuomotor policies that model skills as a joint generative process over action trajectories ( $x$ ) and predicate-belief trajectories ( $z$ ).

Core Formulation

Instead of learning $p(x|o)$ and $p(z|o)$ separately, PACTS learns a coupled distribution:
$(x, z) \sim p_\theta(x, z | o)$
where:

$x$ is the action trajectory over a horizon $H$ .
$z$ is a predicate-belief trajectory, representing a compact trace of soft truth values (confidence scores in $[0, 1]$ ) for a set of predicates $J$ over time.
$o$ is the current observation.

By sampling from this joint distribution, the model produces coherent pairs where the commitment to a specific action mode inherently commits to the corresponding symbolic trace.

Architectural Implementation

Joint Variable: The model defines a joint trajectory variable $y \triangleq [x | z]$ and learns a single conditional generative model $p_\theta(y | o)$ .
Objectives: The framework supports both Denoising Diffusion Probabilistic Modeling (DDPM) and Conditional Flow Matching (CFM).
- DDPM: Trains a noise-prediction network $\epsilon_\theta$ to denoise the joint variable $y$ .
- CFM: Trains a vector field $v_\theta$ to transport noise to data along an optimal transport path.
Backbones: The trajectory generator can be instantiated with either a temporal UNet (1D convolutions) or a Transformer (self-attention over time steps).
Execution: PACTS operates in a receding-horizon manner. At each control cycle, it conditions on the current observation, samples a joint rollout $(x, z)$ , executes a prefix of the action, and re-samples.

Planning-Based Composition

The generated predicate-belief trajectory $z$ serves as an online symbolic interface for planning.

State Estimation: The continuous belief values in $z$ are thresholded (e.g., $>0.5$ ) to produce discrete predicate states $\hat{p}_t$ .
Planning Loop: An off-the-shelf PDDL planner uses these predicted states to:
- Check preconditions and effects.
- Sequence skills to achieve goals.
- Trigger replanning if the predicted effects of a skill are violated (e.g., due to environmental perturbations).
  This enables zero-shot composition, allowing robots to sequence learned skills to achieve novel goals without retraining.

Data Tooling

To facilitate real-world deployment, the authors introduce a skill segmentation and labeling toolkit. This tool converts monolithic demonstration datasets (e.g., LeRobot format) into skill-centric datasets by:

Annotating sparse keyframes with grounded predicates.
Interpolating these into dense per-timestep labels.
Segmenting episodes into contiguous per-skill trajectories based on goal-achieving predicate configurations.

3. Key Contributions

The paper makes three primary contributions:

Joint Generative Formulation: A skill model that treats action trajectories and predicate traces as a coupled generative process, producing coherent action–outcome rollouts within a single model.
Planning-Based Composition: A method for sequencing and monitoring skills using predicate-belief traces as an online interface, enabling zero-shot composition with off-the-shelf planners.
Open-Source Toolkit: A segmentation and labeling toolkit that converts monolithic demonstrations into datasets suitable for joint $(x, z)$ learning.

4. Experimental Results

The authors evaluated PACTS on a 2D compositional benchmark (PushBarrier), two 3D RoboMimic tasks (Kitchen and Coffee Preparation), and a real-world cube packing task.

Single-Policy Performance

Action Quality: In the 2D PushBarrier environment, PACTS achieved perfect or near-perfect task rewards (Max Reward $\approx 1.0$ ) across various backbones (UNet/Transformer) and objectives (DDPM/CFM), outperforming or matching action-only baselines. In 3D tasks, PACTS remained competitive with action-only policies, demonstrating that adding outcome modeling does not degrade control performance.
Predicate Prediction: PACTS consistently outperformed baselines in predicate classification (F1, Recall, Precision, Accuracy). Notably, it surpassed a multi-task baseline with a shared encoder but separate heads, suggesting that joint sampling provides a stronger inductive bias than simple parameter sharing.

Coherence and Composition

Predicate-Action Coherence: Experiments showed that PACTS's predicted belief rollouts closely matched the actual outcomes realized by executing the generated actions (low Cross-Brier and Cross-NLL scores), confirming that the symbolic trace is action-dependent rather than a prior.
Zero-Shot Composition: In simulation and real-world settings, the system successfully composed learned sub-skills to achieve novel goals (e.g., packing only specific colored cubes) that were not demonstrated end-to-end.
Adversarial Recovery: The system demonstrated the ability to detect when a skill's effects were violated (e.g., a door closing mid-execution) via the predicate interface, triggering replanning to recover and complete the task.

5. Significance and Claims

The paper argues that the fundamental limitation in current skill learning is the decoupling of actions from their symbolic outcomes. By modeling skills as joint action–outcome rollouts, PACTS provides a "planner-aligned" interface that is naturally consistent with the policy's behavior.

The authors claim that this approach enables:

Robust Generalization: The ability to sequence skills for unseen task compositions without retraining.
Reliable Monitoring: An online symbolic interface that allows planners to verify execution progress and detect failures.
Inductive Bias: Learning the coupled distribution encourages the policy to represent the specific relationship between action modes and symbolic transitions, improving both control and prediction.

The paper concludes modestly, acknowledging that performance relies on the quality of the predicate set and that the current approach assumes a manually defined symbolic domain. However, it posits that the joint modeling of actions and outcomes is a necessary step toward scalable, reliable, and composable robotic skills.