From Pixels to Predicates: Learning Symbolic World Models via Pretrained Vision-Language Models

This paper proposes a method that leverages pretrained vision-language models to learn compact, abstract symbolic world models from limited visual demonstrations, enabling zero-shot generalization and long-horizon planning for complex robotic tasks across novel objects, environments, and goals.

The Gist

Imagine you are trying to teach a robot how to clean a messy room.

The Old Way (The "Parrot" Approach):
Usually, if you want a robot to learn, you show it a video of a human cleaning. The robot tries to memorize the exact movements: "Move arm left, pick up red cup, move to bin, drop."
This works great if the room looks exactly the same every time. But what if the cup is blue? What if the bin is on the floor instead of the table? What if there's a new object, like a toy, on the table? The robot gets confused. It's like a parrot that can only repeat a song but doesn't understand the lyrics. If you change the tune, the parrot stops singing.

The New Way (The "Translator" Approach - Pix2Pred):
This paper introduces a new method called pix2pred. Instead of just memorizing movements, this method teaches the robot to understand the story of the room using a special "translator" (an AI called a Vision-Language Model).

Here is how it works, step-by-step:

1. The "Translator" (The Vision-Language Model)

Imagine you have a very smart, well-read librarian who can look at a photo and describe it in perfect English.

• The Input: You show the robot a few short videos of a human cleaning.
• The Magic: The "librarian" looks at the images and starts inventing a vocabulary of concepts (called predicates).
  • Instead of just seeing "pixels," the librarian says: "Oh, I see a Table," "That is an Eraser," "The table is Clean," "The bin is Full," "The robot's hand is Empty."
  • The robot doesn't just see shapes; it learns the meaning of the scene.

2. The "Filter" (The Optimization)

The librarian might get a little too excited and suggest 100 different concepts (e.g., "Is the table round?", "Is the table blue?", "Is the table happy?"). Most of these are useless.

• The paper's algorithm acts like a strict editor. It looks at the videos and asks: "Which of these 100 concepts actually helped the human solve the problem?"
• It throws away the fluff and keeps only the important ones, like "Is the table clear?" or "Is the object inside the bin?"
• Now the robot has a tiny, powerful dictionary of rules that actually matter.

3. The "Chess Player" (The Planner)

Now, the robot has a new goal: "Clean the table, but this time the eraser is hidden inside a box."

• Old Robot: "I've never seen an eraser in a box! I don't know what to do!" (It fails).
• Pix2Pred Robot: It uses its dictionary. It thinks:
  1. Goal: Eraser needs to be on the table.
  2. Current State: Eraser is in a box.
  3. Plan: I need to Open Box -> Take Eraser -> Wipe Table.
  4. It checks its rules: "Can I take the eraser? Yes, if my hand is empty."
  5. It builds a plan step-by-step, like a chess player thinking three moves ahead.

The Real-World Test

The researchers tested this on a real robot (a Boston Dynamics Spot dog-robot) and in video game simulations.

• The Challenge: They trained the robot on simple tasks (like wiping a table with one object).
• The Test: They then asked the robot to do complex, new things it had never seen, like:
  • Cleaning a table with five objects instead of one.
  • Wiping a table in a completely different room with different lighting.
  • Retrieving an object from a bin, wiping the table, and putting the object back (a multi-step puzzle).

The Result: The robot succeeded! Because it learned the concepts (like "Empty Hand" or "Full Bin") rather than just copying movements, it could generalize. It was like teaching a child the rules of cooking rather than just showing them how to make one specific sandwich. Once they know the rules, they can make a pizza, a salad, or a sandwich, even if they've never made that specific dish before.

In a Nutshell

pix2pred is a method that uses a smart AI to translate raw camera images into a simple, logical language (like "The cup is on the table"). It then filters out the noise to find the most important rules. This allows a robot to plan its actions logically, solving brand-new problems by combining these simple rules, rather than just blindly copying what it saw before.

It turns a robot from a mindless parrot into a thoughtful problem solver.

Technical Summary

Here is a detailed technical summary of the paper "From Pixels to Predicates: Learning Symbolic World Models via Pretrained Vision-Language Models" (pix2pred).

1. Problem Statement

The core challenge addressed is long-horizon decision-making in complex robotics domains using only low-level sensory inputs (images) and a small number of short-horizon demonstrations.

• The Gap: Traditional imitation learning struggles to generalize to novel goals, object arrangements, or visual backgrounds because it often relies on pattern matching rather than understanding underlying causal dynamics. Conversely, symbolic planning requires manually defined "predicates" (abstract concepts like "is holding" or "is clean") and "operators," which is labor-intensive and brittle.
• The Goal: To learn an abstract symbolic world model directly from raw pixel data and a handful of demonstrations. This model must enable zero-shot generalization to novel tasks, objects, and environments by facilitating efficient search-based planning.

2. Methodology: pix2pred

The proposed method, pix2pred, bridges the gap between low-level pixels and high-level symbolic planning by leveraging Pretrained Vision-Language Models (VLMs) within a program synthesis framework. The process consists of three main stages:

A. Predicate Proposal (The "Invention" Phase)

Instead of manually defining predicates, the system uses a VLM to generate a diverse pool of candidate predicates.

1. Input: The system takes a set of demonstrations (sequences of images and actions) and a small set of initial goal predicates.
2. VLM Prompting: For each demonstration, the VLM is prompted with the sequence of images and the executed actions (e.g., "Pick," "Place"). It is asked to propose ground atoms (specific facts about the scene, e.g., Cooked(patty1)) that change or are relevant to the task.
3. Lifting: These specific ground atoms are automatically "lifted" into typed visual predicates (e.g., Cooked(?p:patty)). The VLM is also prompted to generate synonyms and antonyms to ensure diversity in the candidate pool.
4. Filtering: The system removes syntactically incorrect or irrelevant atoms (e.g., referencing objects not present in the scene).

B. Predicate Subselection & Model Learning

The initial pool of predicates is often large (hundreds) and contains noise. The system must select a compact, useful subset.

1. Hybrid Pool: The visual predicates proposed by the VLM are combined with traditional feature-based predicates (derived from object-centric state data like pose or size, if available).
2. Optimization (Hill-Climbing): An optimization-based algorithm (adapted from Silver et al.) performs a hill-climbing search to select the optimal subset of predicates.
   • Objective: The selection is driven by a planning-efficiency objective. The algorithm iteratively adds predicates that allow the learned symbolic operators to explain the demonstrations while minimizing the complexity of the resulting model.
   • Noise Handling: Since VLMs can be noisy, the operator learning process uses "soft" precondition induction (requiring a predicate to hold in a high percentage of transitions, e.g., 80%, rather than 100%) and prunes operators that fit very few data points.

C. Planning and Execution

Once the symbolic world model (predicates and operators) is learned:

1. State Abstraction: At test time, the VLM evaluates the current visual state to determine the truth values of the selected predicates, creating an abstract symbolic state.
2. Search-Based Planning: A classical planner (e.g., PDDL planner) uses the learned operators to find a sequence of actions (skills) that transitions the abstract state from the initial state to the goal state.
3. Execution: The planner outputs a sequence of low-level skills. Continuous parameters for these skills are sampled using learned generative samplers. The robot executes these skills, and the process can include replanning if the state changes unexpectedly.

3. Key Contributions

• Automated Predicate Invention from Pixels: The paper introduces a novel method to automatically invent semantically meaningful symbolic predicates directly from raw images using VLMs, removing the need for manual predicate engineering.
• Aggressive Zero-Shot Generalization: The learned models generalize to:
  • Novel Objects: Different types and numbers of objects.
  • Novel Goals: Tasks requiring compositions of skills not seen in training (e.g., dumping a bin to retrieve an object, then wiping a table).
  • Novel Environments: Different visual backgrounds and layouts.
• Integration of VLMs with Symbolic Planning: Unlike approaches that use VLMs as direct planners (which often fail on long horizons), pix2pred uses VLMs to construct the world model, which is then used by a rigorous search-based planner. This combines the semantic understanding of VLMs with the reliability of symbolic planning.
• Real-World Validation: The method is validated on a Boston Dynamics Spot robot in real-world environments (Cleanup and Juice domains), demonstrating success with only 6–10 human demonstrations.

4. Experimental Results

The authors evaluated pix2pred across five domains (Kitchen, Burger, Coffee, Cleanup, Juice) in both simulation and the real world.

• Simulation Performance:

  • pix2pred achieved the highest success rates across 4 out of 5 domains compared to baselines like ViLa (VLM-only planning) and various ablations.
  • It significantly outperformed ViLa on long-horizon tasks (e.g., Burger tasks with 15+ steps). ViLa tended to fail when the initial state differed slightly from training (e.g., holding an object at the start), whereas pix2pred learned predicates like HandEmpty to handle these cases.
  • Ablation Studies:
    • Subselection is Critical: Removing the optimization-based subselection (using all VLM-proposed predicates) caused performance to collapse due to overfitting and operator bloat.
    • Visual vs. Feature: Using only visual predicates or only feature-based predicates resulted in lower performance than the hybrid approach.

• Real-World Performance:

  • On the Spot robot, pix2pred successfully solved complex tasks (e.g., clearing a table, wiping it, and putting away an eraser) in novel rooms with different objects.
  • It handled novel goal compositions (e.g., "dump the bin to get the eraser, then wipe, then put the eraser back") which were never explicitly demonstrated, proving the model learned the underlying causal dynamics rather than just mimicking trajectories.

5. Significance and Limitations

Significance:

• Scalability: The method reduces the barrier to deploying robots in new environments by requiring only a few video demonstrations and no manual predicate definition.
• Interpretability: The resulting world models are symbolic and interpretable (e.g., IsEraser, NoObjectsOnTop), allowing for debugging and verification, unlike black-box neural policies.
• Bridging Modalities: It effectively leverages the "common sense" and semantic grounding of large VLMs to solve the "grounding problem" in robotics without sacrificing the planning guarantees of symbolic AI.

Limitations:

• Object Descriptors: The system currently requires manual specification of object descriptors (names/types) for the VLM to ground predicates.
• Full Observability: It assumes objects are fully visible; occlusion can lead to incorrect predicate evaluation.
• Optimization Speed: The hill-climbing optimization for predicate selection can be slow (seconds to minutes) as the number of demonstrations and candidate predicates grows.
• Skill Prerequisites: The method assumes a library of parameterized low-level skills is provided; it does not learn the skills themselves from scratch.

In conclusion, pix2pred represents a significant step toward general-purpose robotics by enabling agents to learn structured, interpretable world models from pixels, allowing them to reason and plan effectively in unseen, complex scenarios.