🤖 AI

Bridging Learned Visual Perception and Symbolic Belief-Space Planning

This paper introduces a novel "VLM-as-probabilistic-grounder" paradigm that captures the uncertainty of visual perception as probability distributions over symbolic states, enabling robust belief-space planning that outperforms existing deterministic approaches in partially observable environments.

Original authors: Guy Azran, Michael Navat, Sarah Keren

Published 2026-09-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Guy Azran, Michael Navat, Sarah Keren

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot trying to tidy up a messy living room. It sees a book on a table and a shelf across the room, but its camera is shaky, and the light is dim. In the real world, robots rarely have perfect vision. They cannot see everything at once; objects hide behind furniture, and sensors often misinterpret what they are looking at. For a robot to act safely and effectively, it must admit what it does not know. If it guesses wrong about where an object is, it might try to grab something that isn't there, or worse, try to place an object it isn't holding. This uncertainty is the biggest hurdle in teaching machines to navigate our complex, unpredictable homes.

For years, researchers have tried to solve this by giving robots a "brain" that can look at a picture and instantly decide what is true. They use powerful artificial intelligence models that understand both images and language. The idea was simple: show the robot a photo, ask it "Is the cabinet open?", and if the model says "yes," the robot treats that as an absolute fact and plans its next move. But in practice, this approach is brittle. When the model makes a mistake—which happens often when objects are hidden or the view is unclear—the robot follows a plan based on a lie. It might spend minutes trying to open a door that is already open, or worse, it might give up entirely because it thinks the task is impossible. The core problem is that these systems treat uncertain guesses as certain facts, leaving no room for error.

A team of researchers at the Technion in Israel has proposed a different way to think about this problem. Instead of forcing the robot to make a single, rigid guess about the world, they taught it to hold multiple possibilities in its mind at once. They call their new system RoVLaP. Rather than asking the artificial intelligence model to say "the book is on the table" or "the book is not on the table," the system asks the model to say how likely each of those scenarios is. It might say there is a 60% chance the book is on the table and a 40% chance it is hidden inside a drawer. By keeping these probabilities alive, the robot can build a "belief" about the world that acknowledges uncertainty. It doesn't just plan for the most likely scenario; it plans for a range of likely scenarios simultaneously.

The researchers tested this idea in a simulated home environment, a digital world filled with virtual furniture, drawers, and objects. They gave the robot tasks that are common in household chores, such as sorting books onto shelves, cleaning out drawers, or locking doors. In these tests, the robot had to rely solely on what its camera could see, without any special knowledge about where hidden objects might be. They compared their new method against two other common approaches: one where the AI model directly tells the robot what to do step-by-step, and another where the model gives a single, fixed description of the room for a standard planner to use.

The results showed a clear advantage for the new approach. In the simulated tests, the standard methods often failed completely when the robot's view was blocked or misleading. For example, in a task where a bowl was hidden inside a closed cabinet, the standard robot would assume the bowl was visible and try to grab it immediately, failing every time. The new system, however, recognized that the bowl might be hidden. It planned a sequence of actions that included opening the cabinet first. This single change allowed the robot to succeed in situations where the other methods failed. On difficult tasks, the new system solved 66.7% of the problems, while the standard "grounder" method solved none. On medium-difficulty tasks, it improved success rates by more than 160% compared to the standard method.

Beyond just solving more tasks, the new system was also more efficient. Because it planned for uncertainty from the start, it made fewer mistakes and had to restart its plans less often. The standard methods often had to stop, realize they were wrong, and try again, wasting time and energy. The new system, by contrast, generated plans that were robust enough to handle the initial confusion without needing to stop and rethink. It moved with a kind of cautious confidence, preparing for the possibility that its first guess might be wrong, and having a backup plan ready just in case.

The researchers also proved that this method is mathematically sound. They showed that if the artificial intelligence model is even slightly better than random guessing, the robot will eventually figure out the true state of the world if it keeps observing and updating its beliefs. The system does not require the model to be perfect; it only needs to be consistently better than a coin flip. Over time, as the robot gathers more visual evidence, its uncertainty shrinks, and its plans become more precise. This provides a safety net: even if the robot starts with a wrong idea, the system is designed to correct itself without crashing or getting stuck.

This work represents a shift in how we build autonomous agents. It moves away from the idea that a robot must know the truth to act, and toward the idea that a robot can act wisely even when it is unsure. By treating the world as a set of possibilities rather than a single fixed reality, the robot becomes more adaptable and reliable. In the simulated homes, this meant the difference between a robot that gives up when it can't see a hidden object and one that patiently opens the cabinet to find it. As robots move from controlled labs into our actual homes, where light changes, objects move, and views get blocked, this ability to plan for the unknown may be the key to making them truly useful companions.

Technical Summary: Bridging Learned Visual Perception and Symbolic Belief-Space Planning via Probabilistic Grounding

Problem Statement

Robotic agents operating in partially observable environments face significant challenges due to uncertainty in perception and action outcomes. Traditional planning frameworks often rely on the Closed World Assumption (CWA) or deterministic symbolic grounding, treating unobserved facts as false or VLM predictions as absolute truths. Recent integrations of Vision-Language Models (VLMs) into planning loops have introduced two paradigms: VLM-as-planner (directly mapping images to action sequences) and VLM-as-grounder (mapping images to deterministic symbolic states for off-the-shelf planners).

The authors argue that both paradigms fail to account for the inherent uncertainty in visual grounding. In domains like household robotics, where objects may be occluded, hidden, or ambiguous, treating VLM outputs as deterministic leads to brittle plans. If a VLM incorrectly predicts a state (e.g., believing a robot is holding an object when it is not, or failing to detect a hidden object), the resulting plan may fail immediately, requiring costly replanning or causing task failure. The core problem addressed is how to leverage VLMs for symbolic state estimation while explicitly modeling and planning under the resulting perceptual uncertainty.

Methodology: Robust Vision-Language Planning (RoVLaP)

The paper introduces RoVLaP, a third paradigm termed VLM-as-probabilistic-grounder. This approach integrates VLMs into a Conformant Probabilistic Planning (CPP) framework. The pipeline consists of three main components:

1. VLM-Based Symbolic State Estimation

Instead of asking the VLM for a deterministic "true/false" label for task-relevant fluents (predicates), RoVLaP queries the VLM for the probability distribution over tokens "true," "false," and "null."

Probabilistic Grounding: For each fluent $f$ , the system extracts the VLM's internal token probabilities. If "null" is the most likely token (indicating insufficient evidence), the probability is set to 0.5. Otherwise, the "true" and "false" probabilities are normalized to define a belief $\beta_f$ for that fluent.
Belief Maintenance: The system maintains a factored belief over individual fluents. It updates this belief over time using logarithmic opinion pooling, allowing the agent to accumulate evidence from sequential observations.
Constraint Handling: To generate a full state belief, the system assumes conditional independence between fluents but enforces domain-specific constraints (e.g., mutual exclusion) derived from invariant discovery tools (like Fast Downward). States violating these constraints are assigned zero probability.

2. Conformant Probabilistic Planning (CPP)

Once a belief state $\beta$ is established, RoVLaP formulates a planning problem under uncertainty.

Most Likely Subset of States (MLSS): The system identifies a subset of symbolic states $b^I$ such that their cumulative probability exceeds a user-defined threshold $\theta$ . This is achieved via a search algorithm (Algorithm 1) that explores states in decreasing order of their unconstrained belief, filtering for those that satisfy domain constraints.
Plan Synthesis: A conformant planner (CPOR) is invoked to find a single plan that is guaranteed to achieve the goal for all states within the selected MLSS. This ensures the plan is robust to the specific uncertainty captured in the belief distribution.

3. Execution and Replanning

The execution loop monitors the alignment between the current belief and the observed outcomes.

Replanning Triggers: The system triggers replanning if:
1. Improbable Plan: The probability of the current plan's success drops below $\theta$ based on new observations.
2. Unsafe Action: An action is detected where preconditions are not satisfied across the entire belief set (preventing "hallucinated certainty").
3. Plan Exhaustion: The plan fails to complete the task.
This mechanism explicitly reasons about belief drift caused by VLM inconsistencies, rather than just reacting to single-step perceptual errors.

Key Contributions

VLM-as-Probabilistic-Grounder: A novel paradigm that extracts per-fluent probabilities from VLMs to maintain an explicit symbolic belief, moving beyond brittle deterministic grounding or direct action generation.
Formalization of Robust Visual Planning (RVP): A principled definition of planning in visual robotic domains that accounts for both perceptual uncertainty (observation-to-state mapping) and partial observability.
The RoVLaP Framework: A complete system that compiles VLM-derived beliefs into Conformant Probabilistic Planning problems, utilizing off-the-shelf planners to generate robust plans.
Theoretical Guarantees: The paper provides conditions (e.g., weak calibration of the VLM and minimum visibility rates) under which the true state is guaranteed to be included in the MLSS, ensuring the existence of a satisficing plan. It also offers safety guarantees regarding the execution of unsafe actions compared to deterministic baselines.

Experimental Results

The authors evaluated RoVLaP on the ViPlan-HH benchmark, a household robotics domain built on iGibson. The evaluation was modified to remove privileged information (ground truth for hidden objects), creating a true partially observable setting.

Comparative Performance: RoVLaP was compared against VLM-as-planner (VLM-P) and VLM-as-grounder (VLM-G).
- Success Rate: RoVLaP achieved significantly higher success rates across all difficulty levels. On "Hard" tasks, VLM-G solved 0% of instances, while RoVLaP solved 66.7%. On "Medium" tasks, RoVLaP improved success by 166.8% over VLM-G.
- Efficiency: RoVLaP required fewer planner calls (1.9 to 10.0 times fewer) and executed fewer actions (1.2 to 6.8 fewer on average) than the baselines, indicating that robust initial planning reduced the need for reactive replanning.
Qualitative Analysis: In scenarios involving hidden objects (e.g., a bowl inside a closed cabinet), VLM-P and VLM-G often failed because they assumed the object was reachable or visible. RoVLaP maintained a probability distribution over the hidden state, generating plans that included opening containers first, thereby succeeding regardless of the object's initial visibility.

Significance and Claims

The paper claims that RoVLaP demonstrates how foundation models can be leveraged for reliable planning under uncertainty without sacrificing the interpretability of symbolic reasoning. By treating VLM outputs as probabilistic evidence rather than deterministic facts, the approach bridges the gap between learned perception and robust decision-making.

The authors emphasize that their method does not require perfect VLMs; rather, it relies on the VLM being "weakly calibrated" (persistently better than random guessing) and the system's ability to accumulate evidence over time. The work highlights that ignoring perceptual uncertainty leads to brittle robotic behavior, whereas explicitly modeling it via belief-space planning enables agents to handle ambiguity, occlusion, and sensor noise effectively in complex, real-world-like environments. The paper concludes by noting that future work will explore active sensing and embedding these methods into broader Task-and-Motion Planning (TAMP) systems.