🤖 AI

Evidence-Gated Task and Motion Planning with Vision-Language Models

This paper proposes Evidence Acquisition and Feasibility Gating (EAFG), a framework that integrates Vision-Language Models with Task and Motion Planning to dynamically acquire visual evidence and gate planning decisions, thereby improving success rates in long-horizon manipulation tasks under partial observability by preventing execution on unverified assumptions.

Original authors: Tsunehiko Tanaka, Matthew Stephenson, Alistair Macvicar, Edgar Simo-Serra

Published 2026-08-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tsunehiko Tanaka, Matthew Stephenson, Alistair Macvicar, Edgar Simo-Serra

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots that move through our homes and kitchens face a fundamental challenge: they must understand not just what to do, but whether they can actually do it. To follow a simple instruction like "make soup," a machine needs to bridge two very different worlds. First, it must grasp the meaning of the words, knowing that soup requires ingredients like chicken and salt, and that these items need to be heated. This is the realm of language and common sense. Second, it must understand the physical world, knowing exactly where the pot is, whether the cabinet door can be opened without hitting a wall, and if the robot's arm can reach the spice jar. For years, researchers have tried to combine these two skills, using advanced computer models that can see and read to help robots plan their actions. However, a persistent problem has remained: what happens when the robot looks at the kitchen and cannot see everything it needs? If a crucial ingredient is hidden inside a closed drawer, or if it simply isn't there at all, the robot might guess based on what it knows from its training data. It might plan to pick up a carrot that doesn't exist, leading to a failed task or a confused machine that keeps trying to grab an empty space.

A team of researchers from Waseda University and Flinders University has proposed a new way to solve this problem, one that forces the robot to look before it leaps. Their approach, called Evidence Acquisition and Feasibility Gating, changes the order of operations. Instead of immediately trying to cook the soup based on a guess, the robot first acts like a curious explorer. It is programmed to generate small, safe tasks designed solely to gather information, such as opening a closed cabinet or moving a box that might be blocking a view. Once the robot performs these actions, it takes a new picture of the scene. A powerful vision-language model then reviews this new visual evidence to decide what comes next. The system asks a simple but critical question: "Do I now have enough proof to plan the cooking task?" If the answer is yes, the robot proceeds to plan the full sequence of cooking steps. If the answer is no, but there are still places to look, the robot goes back to exploring. If the answer is no, and it becomes clear that a required item is missing, the robot stops completely rather than wasting time trying to use something that isn't there.

The researchers tested this method in a simulated kitchen environment designed to mimic the complexities of a real home. The robot was given a single arm and tasked with making chicken soup, a process that involves finding ingredients, using a stove, and managing various containers. They set up three distinct scenarios to see how the new system would perform compared to older methods that do not check for evidence first. In the first scenario, all the necessary items were present and clearly listed in the instructions. Here, the new system performed just as well as the standard approach, proving that adding this extra step of checking does not slow down or confuse the robot when everything is obvious. The real difference appeared in the second scenario, where the instructions were vague. The robot was told to "make chicken soup" without being told to use salt or pepper. Without the new system, the robot rarely succeeded because it didn't know to look for those hidden seasonings. With the evidence-gathering method, the robot successfully opened cabinets, found the salt and pepper, and completed the recipe in forty percent of the trials, a significant improvement over the baseline.

The third scenario tested the system's ability to recognize when a task is impossible. The robot was instructed to make soup with salt, pepper, and a carrot, but the carrot was not in the kitchen. A standard system would often get stuck in a loop, repeatedly trying to find or use the missing carrot, wasting time and energy. The new system, however, learned to stop. By actively searching the likely places where a carrot might be and finding nothing, the robot's internal gate decided to halt. In these tests, the system stopped correctly ninety percent of the time for one of the models and a full one hundred percent for the other, while drastically reducing the number of failed attempts to grab the missing vegetable. This ability to say "I cannot do this" is just as important as the ability to succeed, as it prevents the robot from acting on false assumptions.

The researchers acknowledge that their method is not perfect. It relies on the robot's ability to successfully open doors and move objects to get a better view. If the robot fails to open a cabinet due to a mechanical issue or a physical constraint, it might not find the evidence it needs, even if the object is there. Despite this limitation, the work demonstrates a clear path forward for robots that need to operate in uncertain environments. By inserting a step of active information gathering before the final planning phase, the system ensures that the robot's actions are grounded in what it can actually see, rather than what it merely assumes. This shift from guessing to verifying allows robots to handle the messy, incomplete reality of human spaces more effectively, making them more reliable partners in tasks that range from cooking a meal to organizing a room.

Technical Summary: Evidence-Gated Task and Motion Planning with Vision-Language Models

1. Problem Statement

Robots executing long-horizon manipulation tasks from natural language instructions face a fundamental challenge: they must simultaneously reason about semantic task structures and geometric feasibility. While Vision-Language Models (VLMs) excel at inferring commonsense procedures and generating high-level subgoals, and Task and Motion Planning (TAMP) ensures geometric and kinematic constraints are met, their integration often fails under partial observability.

In environments where goal-relevant objects may be hidden (e.g., inside closed cabinets) or entirely absent, existing VLM-TAMP systems tend to rely on the VLM's prior knowledge or commonsense assumptions. This leads to the generation of subgoals that lack observational support, resulting in execution failures, inefficient repeated attempts to manipulate non-existent objects, or unintended outcomes. Current methods typically assume a fully observable object-oriented state and lack a mechanism to actively verify unobserved objects before committing to a task plan.

2. Methodology: Evidence Acquisition and Feasibility Gating (EAFG)

The authors propose Evidence Acquisition and Feasibility Gating (EAFG), a framework designed to actively gather visual evidence and gate the planning process based on that evidence. The system operates through a pre-planning loop that precedes the generation of the final task plan.

2.1 Core Pipeline

The EAFG pipeline (illustrated in Figure 1 of the paper) functions as follows:

Input: The system receives a natural-language goal ( $G_{eng}$ ).
Evidence Acquisition Loop:
- The VLM generates a sequence of evidence-acquisition subgoals ( $G^{EA}_t$ ). These are strictly exploratory actions (e.g., opening doors, lifting occluding objects) designed to inspect regions where goal-relevant objects might be hidden.
- Constraint: The VLM is restricted to a predefined set of safe, reversible actions. It cannot perform irreversible task actions (e.g., cooking, heating) during this phase.
- Execution: A TAMP module (specifically PDDLStream) attempts to instantiate these subgoals into geometrically feasible action sequences. The robot executes them.
- Observation: Resulting observation images are recorded and compiled into an image collage ( $I_t$ ).
- State Update: The VLM updates an evidence state ( $E_t$ ), which includes the execution history ( $H_t$ ), a textual summary of observed facts ( $O_t$ ), and retained findings ( $F_t$ ). The VLM regenerates these textual components to remove outdated or contradicted descriptions, ensuring the state reflects current reality.
Feasibility Gating:
- After each iteration, the VLM evaluates the evidence state against the goal and outputs a gate status ( $s_t$ $s_{t}$ ):
  - Ready to plan: Sufficient evidence exists; the system exits the loop and proceeds to task planning.
  - Need more evidence: Evidence is insufficient, but unexplored relevant regions remain. The VLM generates new evidence-acquisition subgoals ( $G^{EA}_{t+1}$ ), and the loop continues.
  - Halt: Required objects or conditions cannot be verified (e.g., an object is absent after checking all plausible locations). The system stops execution to prevent planning based on unsupported assumptions.
Evidence-Conditioned Planning:
- If the status is "Ready to plan," the system generates the final task subgoals ( $G_{task}$ ) using the accumulated evidence state ( $E_T$ ) as explicit textual context. This ensures the final plan is grounded in verified observations rather than assumptions.

3. Key Contributions

The paper outlines three primary contributions:

Pre-planning Evidence Acquisition: A mechanism that prompts VLMs to plan information-gathering subgoals to reveal hidden regions and verify the existence of goal-relevant objects before any task execution begins.
Feasibility Gating: A decision-making mechanism that evaluates whether acquired evidence supports feasible task planning. It dynamically requests further observations or halts execution if goal requirements remain unsupported.
Empirical Validation in Cooking Scenarios: The design and evaluation of three specific cooking scenarios (explicitly specified, underspecified, and missing-object goals) demonstrating that EAFG improves grounded task completion and infeasibility detection compared to standard VLM-TAMP baselines.

4. Experimental Results

The authors evaluated EAFG in a kitchen environment using a robot with a single arm, focusing on a "make chicken soup" task. Three settings were tested:

S1 (Explicit-and-Present): The instruction specified all objects (salt, pepper), and they were present.
- Result: EAFG remained competitive with the baseline. For GPT-5.5, Recipe Completion Rate (CR) improved from 0.20 to 0.45, and step completion for specified seasonings reached 1.00.
S2 (Underspecified): The instruction was "make chicken soup" without specifying salt or pepper.
- Result: EAFG significantly outperformed the baseline. Without EAFG, Recipe CR was near zero (0.05 for GPT-5.5). With EAFG, it rose to 0.40 for GPT-5.5 and 0.20 for Gemini-3.5-Flash. The system successfully discovered the unmentioned but necessary objects through exploration.
S3 (Explicit-but-Missing): The instruction required a carrot, which was absent from the environment.
- Result: EAFG drastically improved the system's ability to recognize infeasibility. For GPT-5.5, the Halt Success Rate (SR) increased from 0.45 to 0.90, and the number of attempts to manipulate the missing carrot dropped from 4.00 to 0.55. For Gemini-3.5-Flash, Halt SR reached 1.00 with 0.00 missing-object attempts.

5. Significance and Claims

The paper claims that EAFG addresses a critical gap in long-horizon robot manipulation: the disconnect between semantic inference and physical reality under partial observability. By placing evidence acquisition before task planning, the framework ensures that plans are grounded in verified observations.

The authors state that EAFG:

Improves Grounded Completion: It enables robots to complete recipes by discovering hidden, task-relevant objects that are not explicitly mentioned in the prompt.
Prevents Unintended Outcomes: It reduces repeated, futile attempts to manipulate non-existent objects by halting execution when evidence is insufficient.
Maintains Compatibility: The framework is designed to work with existing VLM-TAMP architectures by appending the evidence state as textual context to the planner prompt.

The authors acknowledge limitations, noting that EAFG relies on the TAMP module to successfully execute exploratory subgoals; if TAMP fails to find a feasible path for an evidence-gathering action, the system may lack the necessary information to make a correct gating decision. However, they posit that as underlying planning and feedback mechanisms improve, the robustness of EAFG will correspondingly increase.