💻 computer science

SG-CoT: An Ambiguity-Aware Robotic Planning Framework using Scene Graph Representations

This paper introduces SG-CoT, a two-stage robotic planning framework that leverages scene graph representations to enable large language models to detect and resolve environmental ambiguities through iterative querying, thereby significantly improving planning accuracy and success rates in both single-agent and multi-agent scenarios.

Original authors: Akshat Rana, Peeyush Agarwal, K. P. S. Rana, Amarjit Malhotra

Published 2026-03-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Akshat Rana, Peeyush Agarwal, K. P. S. Rana, Amarjit Malhotra

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a robot chef in a busy kitchen. Your boss (the user) gives you an instruction: "Get me the red cup."

In a perfect world, there is only one red cup on the counter. You grab it, and you're done. But what if there are three red cups? Or what if there are no red cups at all? Or what if your boss says, "Get me something to eat," and you have a fridge full of options?

This is the problem of ambiguity. Current robot brains (powered by Large Language Models, or LLMs) are like very confident but slightly absent-minded interns. If they don't see a red cup, they might just guess, "Oh, I'll grab that blue one, it's close enough!" or they might grab the wrong red cup. This leads to mistakes, broken dishes, or unsafe actions.

The paper you shared introduces a new system called SG-CoT (Scene Graph-Chain-of-Thought) to fix this. Here is how it works, explained with simple analogies:

1. The Problem: The "Confident Guess"

Older robot planners are like someone trying to solve a puzzle while wearing blindfolds, but they are too confident. They hear "Get the red cup," and even if they can't see the cups clearly, they just pick one and hope for the best. They don't know how to say, "Wait, I see three red cups; which one do you want?"

2. The Solution: The "Organized Librarian" (The Scene Graph)

SG-CoT changes the game by giving the robot a Scene Graph.

Think of the environment (the kitchen) not just as a blurry photo, but as a highly organized digital library card catalog.

Instead of just seeing "a cup," the system builds a structured map.
It knows: "Object #1 is a red cup, it is on the table, and it is next to a blue bowl."
It knows: "Object #2 is a red cup, it is inside the cabinet."

This map is built using a special camera brain (Vision-Language Model) that describes everything in the room with precise details and relationships.

3. The Process: The "Detective's Notebook" (Chain-of-Thought)

Once the map is built, the robot doesn't just guess. It uses a Chain-of-Thought process, which is like a detective writing down their thoughts in a notebook before making an arrest.

Here is the step-by-step loop SG-CoT uses:

Read the Clue: The robot gets the instruction ("Get the red cup").
Check the Map: Instead of guessing, the robot asks its "Librarian" (the Scene Graph): "Hey, show me all the red cups."
The "Aha!" Moment: The Librarian replies: "I found two red cups: one on the table, one in the sink."
Realize the Ambiguity: The robot's detective brain says, "Oh no! The boss didn't say which one. If I grab the wrong one, I fail."
Ask for Help: Instead of grabbing blindly, the robot stops and asks the human: "I see two red cups. Do you mean the one on the table or the one in the sink?"

4. Why This is a Big Deal

The paper tested this in two scenarios:

The Solo Robot: When a single robot faces a confusing room, SG-CoT was 4% to 15% more successful than previous methods because it stopped guessing and started asking.
The Team of Robots: Imagine two robots working together, but they can only see half the room each (like two people looking through different windows). If Robot A needs an object it can't see, it uses SG-CoT to ask Robot B: "Hey, can you check your side of the room for a red cup?" This teamwork solved problems that other robots couldn't handle at all.

The Bottom Line

SG-CoT is like giving a robot a magnifying glass and a notepad.

Old Robots: "I think I see a cup. I'll grab it!" (Mistake happens).
SG-CoT Robot: "Let me check my map. I see three cups. Which one is the boss talking about? Let me ask." (Success!).

It turns a robot that blindly follows orders into a thoughtful partner that knows when to pause, check the facts, and ask for clarification to get the job done right.

1. Problem Statement

Large Language Models (LLMs) have shown promise in robotic planning but suffer from a critical limitation: ambiguity. Robots often face situations where instructions are underspecified by the user (e.g., "Pick up the cup" when multiple cups exist) or where the environment state creates ambiguity (e.g., the target object is missing or occluded).

Current Limitations: Existing methods often rely on uncertainty estimation or simple vision-language models (VLMs) that output high-level object lists. These approaches struggle to detect ambiguities rooted in the environment state (e.g., spatial relationships, specific attributes) rather than just the text of the instruction. Consequently, LLMs often act overconfidently, leading to unsafe or suboptimal actions, or fail to ask the correct clarification questions.
Goal: Develop a framework that enables LLM-based planners to detect, diagnose, and resolve ambiguities arising from both user instructions and environmental states by grounding reasoning in a structured representation of the world.

2. Methodology: SG-CoT Framework

The authors propose Scene Graph-Chain-of-Thought (SG-CoT), a two-stage, iterative framework that integrates structured scene graphs with LLM reasoning.

A. Scene Graph Generation

Instead of relying on unstructured text descriptions, the environment is converted into a Scene Graph ( $G = (V, E)$ ):

Nodes ( $V$ ): Represent detected objects with specific attributes (e.g., color, size, state like is_open).
Edges ( $E$ ): Represent semantic and spatial relationships between objects (e.g., left_of, inside_of).
Construction: A Vision-Language Model (VLM) is prompted to generate this graph in a single pass from the image observation and a list of detected objects. The output is strictly formatted as a JSON object to ensure deterministic parsing.

B. Iterative Reasoning Process

The LLM does not generate a plan in a single pass. Instead, it follows an iterative loop (Observation $\to$ Reasoning $\to$ Retrieval $\to$ Decision):

Initialization: The LLM receives the user instruction, the list of detected objects, and the scene graph schema.
Reasoning & Tool Use: The LLM generates a thought trace ( $T_t$ $T_{t}$ ). If it determines more information is needed, it calls a retrieval API function ( $f_t$ $f_{t}$ ) to query the scene graph.
- retrieve_node: Fetches objects based on attributes (e.g., "Find all red bowls").
- retrieve_edge: Fetches relationships (e.g., "What is inside the red bowl?").
Grounding: The results ( $R_t$ ) are appended to the conversation history, allowing the LLM to refine its understanding based on factual data from the graph rather than hallucination.
Termination: The loop continues until the LLM has sufficient context to either:
- Execute a definitive action sequence (e.g., pick_and_place).
- Generate a targeted clarification question identifying the specific source of ambiguity (e.g., "Did you mean the yellow block or the blue block?").

3. Key Contributions

SG-CoT Framework: A novel architecture combining scene graphs with iterative Chain-of-Thought reasoning, allowing LLMs to actively query environmental data to resolve ambiguities.
Formal Ambiguity Classification: The paper categorizes ambiguities into three distinct types, addressing a gap in prior literature:
- Underspecification: Missing referential grounding in the instruction.
- Environment-Induced: Multiplicity (multiple valid targets) or Absence (target missing).
- Observation-Level: Partial observability in multi-agent systems.
Evaluation Metric: Introduction of the Correct Question Rate (CQR), a metric specifically designed to measure whether the robot identifies the correct source of ambiguity and asks the appropriate clarification question.

4. Experimental Results

The framework was evaluated in simulated tabletop environments (single-agent) and multi-agent settings (partial observability), comparing against baselines like CLARA, Inner Monologue, and ProgPrompt.

Single-Agent Performance:
- SG-CoT achieved a Success Rate (SR) of 99% for underspecified instructions and 86.5% for environment-level ambiguities (using Gemini-2.5-Flash).
- It outperformed the best baseline (Inner Monologue) by 10% in CQR and 4% in SR for single-agent tasks.
- Ablation Studies confirmed that iterative retrieval is crucial; providing the full graph at once or removing edge relationships significantly degraded performance.
Multi-Agent (Partial Observability):
- In tasks requiring inter-robot communication (e.g., Stack and Pass), SG-CoT achieved a 75% Success Rate (Gemini), a 15% improvement over the next best baseline.
- The framework successfully enabled robots to query each other for missing observations rather than hallucinating object locations.
Efficiency: While SG-CoT incurs higher token usage and latency due to iterative calls, the number of reasoning iterations does not scale linearly with the total number of objects in the scene, only with the complexity of the specific query.

5. Significance and Conclusion

Robustness: SG-CoT shifts robotic planning from "guessing" based on text to "verifying" based on structured visual data. This significantly reduces hallucinations and unsafe actions in ambiguous scenarios.
Human-Robot Collaboration: By enabling robots to pinpoint the exact reason for ambiguity and ask specific questions, the framework facilitates more natural and efficient human-robot interaction.
Scalability: The approach demonstrates that integrating structured representations (scene graphs) with LLMs is a viable path toward generalizable robotic planning, even in complex, partially observable multi-agent environments.

Limitations: The current system relies on VLMs for initial graph construction, which can introduce errors (hallucinated edges). Additionally, the method is currently simulation-based, and token costs increase with scene complexity. Future work aims to extend this to real-world embodied AI tasks.