💻 computer science

Semantic Planning with Large Language Models and Vision-Language Models for Collaborative Robots: A Review

This survey reviews semantic planning for collaborative robots by formalizing the challenge as a constrained grounding problem, proposing a taxonomy of architectures that balance capability with inspectability, and outlining a roadmap for enhancing reliability through improved uncertainty estimation, plan verification, and recovery mechanisms.

Original authors: Osama Ali Khan, Abuzar Ghaffari

Published 2026-09-01

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Osama Ali Khan, Abuzar Ghaffari

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the factories of the past, robots lived behind fences. They worked in isolated cells, separated from humans by steel barriers and safety gates. If a robot made a mistake, the fence kept the error contained, and the robot's programming was a fixed, predictable thing. But the new generation of robots, known as collaborative robots, is designed to work right alongside people, sharing the same space without a fence. This shift moves the burden of safety from a physical barrier to the robot's own mind. Instead of following a rigid script, these machines are now being taught to understand natural language and visual cues, allowing a human to simply say, "Pick up that blue part," or "Help me assemble this," and expect the robot to figure out the rest. This capability relies on powerful computer systems that can read text and see images, turning vague human requests into a sequence of physical actions. Yet, as these machines become more conversational and capable, a critical question arises: can we trust them to do exactly what we mean, especially when the world is messy and unpredictable?

A new review of the field, led by researchers at the Southern University of Science and Technology and the University of Engineering and Technology Peshawar, examines the current state of this technology. The authors argue that while these robots are becoming remarkably good at understanding language and vision, they are failing at a more fundamental level: they cannot reliably connect their words to the physical reality around them. The problem is not that the robots are stupid; it is that they are too confident in their own guesses. When a human asks a robot to move a specific object, the robot's internal system might generate a perfect-sounding plan to do so. However, if that object is actually hidden behind a box, or if the robot's arm cannot physically reach it, the robot often proceeds anyway. It does not stop to check if its plan makes sense. Instead, it executes the motion, fails silently, and moves on to the next step, leaving the human to wonder why the task was not completed. This "silent failure" is the central danger the paper identifies. Unlike older robots that would simply stop and error out when they encountered a problem, these new systems generate fluent, logical-sounding plans that are physically impossible, and they offer no warning that they are about to fail.

The researchers analyzed dozens of recent systems that use large language models and vision-language models to control robots. They found that the field has been moving in a direction that makes these problems worse. Over the last few years, the architecture of these systems has shifted from generating text that humans could read and check, to generating code, and finally to generating direct action commands. While this change has made the robots faster and more general-purpose, it has also removed the "middleman" steps where a human or a safety system could inspect the plan before it was executed. The most capable robots today are the least transparent. They act as a black box, taking an instruction and producing a movement without revealing the reasoning in between. The review highlights that this lack of transparency is a major barrier to safety. If a robot cannot explain why it is doing something, or if it cannot admit when it is unsure, it cannot be a true partner. It becomes a hazard that happens to be polite.

To understand the scope of the issue, the authors broke down the requirements for a truly safe and reliable collaborative robot. They identified five key conditions that current systems struggle to meet simultaneously. First, the robot must be "grounded," meaning it must verify that the objects it is talking about actually exist in the room and are where it thinks they are. Second, the plan must be "feasible," ensuring the robot's arm can physically perform the motion without hitting its own limits. Third, it must be "safe," adhering to strict rules about speed and force when near a human. Fourth, the robot's actions must be "legible," meaning a human partner should be able to guess what the robot is trying to do just by watching its movements. Finally, and perhaps most importantly, the robot must be able to "abstain," or admit when it is not confident enough to act, and ask for help. The review found that while some systems are good at one or two of these, almost none are good at all of them. Most systems are excellent at generating a plan that sounds right but fail to check if it is actually possible, and they rarely have a mechanism to stop and ask a human for clarification when things get uncertain.

The paper proposes a way to fix this by changing how we think about reliability. The authors suggest that it is better to have a robot that is slightly less accurate but very good at detecting its own mistakes than a robot that is highly accurate but blind to its errors. They argue that in a long sequence of tasks, a single undetected error can ruin the entire job. Therefore, the priority should be on building systems that constantly check their own work. If a robot sees that an object is missing or that a grasp is unlikely to succeed, it should stop and ask the human for help rather than trying to force the action. This approach requires a shift in how these systems are built. Instead of letting the artificial intelligence model make all the decisions, the model should act as a generator of ideas, while a separate, simpler, and more reliable system acts as a verifier to check those ideas before they are executed. This "verifier-in-the-loop" approach would allow the robot to keep its ability to understand complex language while ensuring that every action is physically safe and grounded in reality.

The researchers also point out that the way these robots are tested is part of the problem. Most studies evaluate performance in simulations or on simple, clean tabletop tasks where everything is perfectly visible and arranged. These tests do not capture the chaos of a real factory floor, where lighting changes, objects are cluttered, and humans move unpredictably. The review calls for a new standard of testing that includes real humans working alongside the robots in real environments. Without this kind of rigorous testing, the field cannot know if the robots are truly ready for the real world. The authors emphasize that the technology to build safe, collaborative robots already exists; what is missing is the discipline to verify it properly and the willingness to prioritize safety and transparency over raw speed and generalization.

Ultimately, the paper concludes that the future of collaborative robotics depends on humility. The most advanced robots today are often the most overconfident, generating fluent plans that ignore physical constraints. The path forward requires building systems that know when they do not know. A robot that can pause, look at a human, and say, "I am not sure I can reach that, can you help?" is a true collaborator. A robot that blindly executes a flawed plan is a hazard, no matter how well it speaks. The review serves as a roadmap for engineers and researchers to move beyond the current limitations, urging them to build systems that are not just smart, but also careful, transparent, and safe enough to work side-by-side with people.

Technical Summary: Semantic Planning with Large Language Models and Vision-Language Models for Collaborative Robots

Problem Statement

The paper addresses a critical reliability gap in collaborative robotics (cobots). While Large Language Models (LLMs) and Vision-Language Models (VLMs) enable robots to translate natural language instructions into task plans, these plans often lack grounding in physical reality, feasibility regarding robot kinematics, and safety regarding human partners.

The core issue is silent failure. Unlike classical symbolic planners that fail explicitly when preconditions are unmet, foundation-model-based planners generate fluent, semantically coherent action sequences that may contain non-existent referents, ungraspable objects, or unsafe motions. These errors remain undetected until execution, often too late for recovery. This is particularly damaging over long-horizon tasks where local errors propagate, compromising the entire task without triggering safety mechanisms. The paper argues that current systems fail to satisfy five essential conditions for reliable collaboration: groundedness, feasibility, safety, legibility, and calibrated abstention.

Methodology

The authors conducted a systematic review of literature from January 2020 to August 2026, sourcing 150 references from major databases (IEEE Xplore, Scopus, arXiv, etc.). The methodology involved:

Taxonomy Construction: Categorizing existing architectures based on the depth at which semantics enter the planning and control hierarchy.
Formal Definition: Defining "Collaborative Semantic Planning" as a constrained grounding problem (Definition 1) that maps instructions and observations to robot skills while satisfying specific predicates (P1–P5).
Reliability Decomposition: Developing a quantitative model (Proposition 1) to separate per-step accuracy from detection quality, analyzing the conditions under which improving detection dominates improving accuracy.
Gap Analysis: Comparing current systems against the five defined conditions and existing safety standards (ISO/TS 15066, ISO 10218).

Key Contributions

The paper makes seven primary contributions:

Formal Definition of Collaborative Semantic Planning: The authors define the problem as a mapping $\Pi: \mathcal{L} \times \mathcal{O} \times \mathcal{H} \to (\mathcal{A} \times \Theta)^n$ that must satisfy five conditions:
- Groundedness (P1): Referents must correspond to perceived entities, not just language priors.
- Feasibility (P2): Actions must be kinematically and dynamically reachable.
- Safety (P3): Trajectories must satisfy separation and force constraints relative to predicted human reach.
- Legibility (P4): The robot's intent must be inferable by the human partner.
- Calibrated Abstention (P5): The planner must emit calibrated confidence scores and abstain from acting when confidence is low.
Architectural Taxonomy: The field is organized into five classes based on where semantics enter the control chain:
- LLM-centric: High horizon, low grounding (e.g., SayCan).
- VLM-centric: High grounding, short horizon (e.g., CLIPort).
- Hybrid: Attempts to combine both but struggles with modality disagreement.
- Closed-loop perception-action: Integrates execution feedback but often lacks calibrated verification.
- Multi-agent: Coordination via dialogue, introducing compounded hallucination risks.
- Key Finding: There is a trade-off where the most capable (generalizable) systems are the least inspectable.
Reliability Decomposition (Proposition 1): The paper derives a mathematical condition showing that a planner with lower per-step accuracy ( $p'$ ) but high detection recall ( $r$ ) and recovery success ( $\rho$ ) outperforms a highly accurate ( $p$ ) but non-detecting planner if:
$r\rho > \frac{p - p'}{1 - p'}$
This proves that for long horizons, investing in detection and recovery is often more effective than chasing marginal gains in per-step accuracy.
Failure Analysis: The review identifies seven root causes of failure, including the symbol-grounding gap, open-world perception brittleness, lack of calibrated uncertainty, and the mismatch between token-level generation and continuous control.
Five Practical Solutions: The authors propose five implementable interventions:
- Verifier-in-the-loop Neuro-symbolic Planning: Using a symbolic layer to veto LLM proposals before execution.
- Calibrated Uncertainty with Principled Abstention: Using conformal prediction to trigger human intervention when confidence is low.
- Grounded Execution Monitoring: Multi-channel detection (vision, force, proprioception) with a structured recovery hierarchy.
- Human-Aware Personalized Planning: Incorporating human reach sets and legibility into the planning state with bounded memory.
- Standardized Benchmarking: A protocol requiring the reporting of failure logs, false-positive rates, and human-in-the-loop metrics.
Evaluation Gap Identification: The paper highlights that current literature overwhelmingly reports aggregate task success rates while omitting critical metrics like detection recall, false-positive rates, intervention rates, and legibility.
Positioning: The review explicitly distinguishes itself from prior surveys by focusing on the human-robot dyad, treating failure detection as central, and addressing safety standards and quantitative reliability models.

Results and Findings

Architectural Trend: Between 2022 and 2026, architectures have shifted from emitting text to emitting code to emitting action tokens. While generalization has improved, inspectability has decreased, moving in the opposite direction of capability.
Silent Failure: Most surveyed systems lack mechanisms to detect semantic errors (e.g., wrong object, wrong intent) before execution. Reactive safety layers (ISO/TS 15066) only prevent physical injury after a plan has already failed semantically.
Human Factors: Current semantic planning literature largely ignores human intent prediction and legibility, treating humans merely as obstacles to avoid rather than partners to collaborate with.
Data Scarcity: There is a severe lack of physical hardware evaluations involving human participants. Most benchmarks are simulation-only or use toy tasks that do not reflect real-world clutter and occlusion.
Quantitative Insight: Under the independent-step model, the threshold for detection quality to dominate accuracy is independent of plan length. However, in reality (positive dependence), early detection is even more valuable.

Significance and Claims

The paper claims that the primary barrier to deploying LLM-driven collaborative robots is not a lack of capability, but a lack of reliability and verifiability.

The "Silent Failure" Problem: The most significant finding is that foundation models fail silently. A plan can be linguistically perfect but physically impossible or unsafe, and without explicit verification, the robot executes it.
The Necessity of Abstention: The authors argue that the ability to say "I don't know" (calibrated abstention) is more critical than the ability to guess correctly. A robot that knows when it is uncertain is a collaborator; one that does not is a hazard.
Engineering vs. Research: The paper asserts that the solutions to these problems (verification, calibration, monitoring) do not require new research breakthroughs but rather the integration of existing methods (neuro-symbolic verification, conformal prediction, force sensing) into a cohesive architecture.
Future Direction: The field must shift from optimizing for aggregate task success to optimizing for detectability and recovery. The authors call for a new generation of architectures that treat their own outputs as hypotheses to be verified, supported by standardized benchmarks that include human participants and rigorous failure reporting.

The paper concludes that without addressing the gap between semantic planning and physical grounding, and without establishing composite safety cases that account for stochastic planners, these systems will remain confined to laboratories. The path forward lies in architectures that prioritize inspectability and uncertainty quantification alongside generalization.