ViPlan: A Benchmark for Visual Planning with Symbolic Predicates and Vision-Language Models

The paper introduces ViPlan, the first open-source benchmark comparing VLM-grounded symbolic planning against direct VLM planning across Blocksworld and household robotics domains, revealing that symbolic approaches excel in visually grounded tasks while direct planning outperforms in scenarios requiring linguistic knowledge, alongside findings that Chain-of-Thought prompting offers no consistent benefit.

The Gist

Imagine you are trying to teach a robot to clean your house or stack blocks. You have two main ways to give it instructions, and a new study called ViPlan just put them head-to-head to see which one actually works better.

Here is the breakdown of the paper in simple terms, using some everyday analogies.

The Two Approaches: The "Intuitive Chef" vs. The "Strict Accountant"

The researchers compared two different ways robots (powered by AI) can plan their actions:

1. VLM-as-Planner (The Intuitive Chef):

   • How it works: You show the robot a picture of the room and say, "Clean the dishes." The robot looks at the picture and immediately guesses, "Okay, I'll grab the fork, then the plate, then the cup." It makes a plan on the fly, relying on its general knowledge of how the world works.
   • The Analogy: This is like a Chef who has cooked a million meals. They don't need a recipe book; they just look at the ingredients and intuitively know what to do next. They are fast and creative but might miss a tiny detail if they aren't looking closely.

2. VLM-as-Grounder (The Strict Accountant):

   • How it works: The robot doesn't guess the whole plan. Instead, it acts like a translator. It looks at the picture and answers very specific "Yes/No" questions to a strict logic engine (a symbolic planner).
     • Question: "Is the bowl on the table?" -> Answer: "Yes."
     • Question: "Is the bowl reachable?" -> Answer: "Yes."
     • Once it confirms all the facts, the "Accountant" (the logic engine) calculates the perfect, step-by-step plan.
   • The Analogy: This is like a Strict Accountant who refuses to move a single dollar until they have verified every single receipt. They are incredibly precise and won't make logical errors, but they are slow and can get confused if the receipts are blurry or if they can't see everything.

The Test: Two Different Kitchens

The researchers created two different "kitchens" (environments) to test these robots:

1. The Block Tower (ViPlan-Blocksworld):

   • The Setup: A simple world with colored blocks stacked in columns. It's very clear, organized, and everything is visible.
   • The Winner: The Strict Accountant (VLM-as-Grounder) won easily.
   • Why? In this clean world, the "Yes/No" questions are easy to answer. The Accountant's precision shines because there are no hidden tricks. The Intuitive Chef got confused because they tried to guess too much without checking the facts first.
   • The Score: The Accountant solved 46% of the tasks, while the Chef only managed 9%.

2. The Messy House (ViPlan-Household):

   • The Setup: A simulated robot arm in a messy house. There are drawers, cabinets, and objects hidden behind others. You can't see everything at once.
   • The Winner: The Intuitive Chef (VLM-as-planner) won by a landslide.
   • Why? In a messy house, you can't see everything. The Accountant gets stuck because it can't verify every single fact (e.g., "Is the spoon inside the closed drawer?"). It keeps asking questions it can't answer and freezes. The Chef, however, uses its "common sense" (linguistic knowledge) to guess, "Okay, the spoon is probably in the drawer, so I'll open it." It fills in the gaps with logic.
   • The Score: The Chef solved 34% of the tasks, while the Accountant only managed 5%.

The "Thinking Aloud" Experiment (Chain-of-Thought)

The researchers also tried a popular trick called Chain-of-Thought (CoT), where they asked the AI to "think out loud" before answering.

• The Result: It didn't help much. In fact, for the "Intuitive Chef," it often made things worse.
• The Analogy: It's like asking a nervous student to explain every single step of their math homework before writing the answer. Instead of solving the problem, they get stuck in a loop of overthinking, run out of time (or "token budget"), and fail to finish. The study found that current AI models aren't great at "thinking" their way through complex visual puzzles; they just get confused by their own words.

The Big Takeaway

This paper teaches us that there is no single "best" robot brain.

• If you are in a clean, predictable, and fully visible world (like a factory assembly line), you want the Strict Accountant. You need someone who checks every fact to ensure nothing goes wrong.
• If you are in a messy, unpredictable, and partially hidden world (like a real home), you want the Intuitive Chef. You need someone who can use common sense to guess what's happening even when they can't see it all.

The paper concludes that we need to stop trying to force one method to do everything. Instead, we need to build systems that know when to be a precise accountant and when to be a creative chef. Until then, our robots will keep struggling to clean up our messy houses!

Technical Summary

1. Problem Statement

The integration of Large Language Models (LLMs) and Vision-Language Models (VLMs) into automated planning is a promising avenue for creating verifiable, grounded agents. However, a critical gap exists in the literature: there is no rigorous, open-source benchmark to compare two dominant paradigms:

1. VLM-as-Planner: The VLM directly selects actions or generates plans from visual inputs, relying on implicit world models and linguistic priors.
2. VLM-as-Grounder: The VLM acts as a perception module, translating visual observations into symbolic predicates (e.g., PDDL) for a classical symbolic planner to solve.

Current research lacks a unified environment to evaluate these approaches under matched conditions. Furthermore, it is unclear when one approach outperforms the other, particularly regarding the trade-off between precise visual grounding and the ability to leverage linguistic reasoning in ambiguous environments.

2. Methodology

The ViPlan Benchmark

The authors introduce ViPlan, the first open-source benchmark designed to compare VLM-as-planner and VLM-as-grounder methods. It consists of two distinct visual domains with complementary challenges:

• ViPlan-Blocksworld (ViPlan-BW): A photorealistic visual variant of the classic Blocksworld planning problem.
  • Characteristics: Fully observable, abstract, and deterministic.
  • Challenge: Requires precise state tracking and logical consistency. Linguistic priors offer little advantage over perceptual input.
  • Setup: 25 procedurally generated tasks across three difficulty splits (Simple, Medium, Hard).
• ViPlan-Household (ViPlan-HH): A simulated robotics environment based on iGibson 2.0 using the Fetch robot.
  • Characteristics: Partially observable, realistic, and stochastic. Involves navigation and manipulation (e.g., cleaning, organizing).
  • Challenge: Requires handling occlusion, object ambiguity, and leveraging linguistic knowledge to infer hidden states.
  • Setup: 25 tasks across three difficulty splits, utilizing PDDL domains translated from the BEHAVIOR-100 suite.

Method Classes Evaluated

The benchmark evaluates eight method implementations across 21 open-source VLMs (selecting the top 5 for final benchmarking):

1. VLM-as-Planner:

   • Plan: Generates a full plan (JSON) based on the current image and goal.
   • Action: Selects only the next immediate action.
   • CoT Variants: Both Plan and Action variants include Chain-of-Thought (CoT) prompting to encourage reasoning before output.
   • Mechanism: Closed-loop planning where the VLM replans after every action execution.

2. VLM-as-Grounder:

   • Ground: Uses the VLM to answer Yes/No questions about predicate truth values (e.g., "Is the bowl on the table?") to initialize the symbolic state and verify action preconditions/effects.
   • Ground + CoT: Augments the grounding process with reasoning traces.
   • Ground + Mem: Augments the VLM context with memory of previous grounding failures to correct errors during replanning.
   • Mechanism: A symbolic planner (Fast Downward) generates the plan; the VLM acts as a "perception oracle" to ground the state and detect execution failures.

Model Selection

The authors evaluated 21 open-source models (ranging from ~7B to 78B parameters, including InternVL, Qwen, Gemma, and LLaVA families) plus three closed-source models (GPT-4.1, GPT-5.2) for reference. The top 5 performing models were selected for the main benchmark to ensure reproducibility and manage computational costs.

3. Key Contributions

1. First Open-Source Visual Planning Benchmark: ViPlan enables the first direct comparison between symbolic-grounded and direct VLM planning approaches under matched protocols.
2. Dual-Domain Design: By contrasting ViPlan-BW (abstract, fully observable) with ViPlan-HH (realistic, partially observable), the benchmark isolates the specific strengths and weaknesses of each planning paradigm.
3. Comprehensive Evaluation: The study evaluates 21 VLMs and 8 method variants, providing a granular view of performance across model sizes, architectures, and reasoning strategies.
4. Failure Analysis: The authors provide a qualitative analysis of failure modes, distinguishing between grounding hallucinations, planning inconsistencies, and parsing errors.
5. Open-Source Release: The code, environments, and data are publicly available to facilitate future research.

4. Key Results

Domain-Specific Performance Trade-offs

• In ViPlan-BW (Abstract/Full Observability):
  • VLM-as-Grounder significantly outperforms VLM-as-Planner.
  • Result: Grounder methods solved 46% of tasks vs. 9% for Planners.
  • Reasoning: The task requires precise state tracking. The symbolic planner ensures logical consistency, while the VLM only needs to answer simple, local visual questions. Direct planners struggle to maintain an implicit state model over multiple steps.
• In ViPlan-HH (Realistic/Partial Observability):
  • VLM-as-Planner significantly outperforms VLM-as-Grounder.
  • Result: Planner methods solved 34% of tasks vs. 5% for Grounders.
  • Reasoning: Partial observability and object ambiguity make the "grounding" process error-prone. Small errors in predicate verification compound, leading to dead-ends. Planners leverage linguistic priors to propose plausible actions despite visual ambiguity, bypassing the need for perfect state grounding.

Impact of Chain-of-Thought (CoT)

• No Consistent Benefit: CoT prompting failed to consistently improve performance across most methods.
• Negative Impact: In the Plan + CoT variant for ViPlan-HH, performance dropped significantly (from 39% to 23%).
• Cause: Models often entered repetitive reasoning loops, exhausting the token budget (1024 tokens) before generating a valid plan. This suggests current VLMs struggle to maintain coherent reasoning chains for complex, multi-step visual planning tasks.

Failure Modes

• Planners: Fail due to "state drift" (predicting non-executable actions based on incorrect internal state) or formatting/parsing errors.
• Grounders: Fail due to grounding hallucinations.
  • False Negatives: Incorrectly judging a valid precondition as unsatisfied (leading to dead-ends).
  • False Positives: Incorrectly judging an invalid precondition as satisfied (leading to infinite loops of failed actions).
  • Compounding Errors: Even with high per-predicate accuracy (>90%), the accumulation of errors over 10+ steps in a single task drastically reduces the overall success rate.

5. Significance and Conclusion

ViPlan demonstrates that there is no "one-size-fits-all" solution for visual planning.

• Symbolic Grounding is superior for tasks requiring strict logical consistency and precise state tracking in fully observable environments.
• Direct Planning is superior for tasks requiring robustness to ambiguity and the ability to leverage high-level semantic knowledge in partially observable, real-world scenarios.

The paper highlights that current VLMs still lack the robust visual reasoning capabilities required for long-horizon planning, particularly when relying on CoT. The benchmark serves as a critical tool for diagnosing these limitations and guiding the development of hybrid agents that can dynamically switch between or combine these approaches based on the environment's observability and complexity.