Technical Summary: Grounded Scene-graph Reasoning (GSR) for Embodied Manipulation

Problem Statement

Despite rapid advancements in embodied learning, robotic agents continue to struggle with long-horizon manipulation tasks that require maintaining spatial consistency, causal dependencies, and goal constraints. Current approaches, particularly end-to-end Vision-Language-Action (VLA) models, often implicitly embed task reasoning within high-dimensional latent representations. This design entangles task logic with perceptual variability (e.g., object appearance), making models overly sensitive to environmental changes and limiting their ability to generalize to new tasks or out-of-domain settings. While classical Task and Motion Planning (TAMP) frameworks offer structured solutions, they typically rely on predefined symbolic representations that are difficult to scale to open-ended, high-variability environments. The core challenge addressed is how to enable embodied agents to leverage structured intermediate representations to enhance reasoning capabilities for general manipulation tasks without sacrificing scalability.

Methodology

1. Grounded Scene-graph Reasoning (GSR) Framework

GSR proposes a paradigm shift from direct perception-to-action mapping to explicit reasoning over semantically grounded scene graphs.

• World Representation: Instead of raw RGB-D inputs, the system constructs a 3D scene graph $M_{sg} = (O_t, E_t)$, where $O_t$ represents objects (modeled with functional keypoints and articulated components) and $E_t$ encodes spatial relations (e.g., on, inside, adjacent).
• Architecture: GSR utilizes a fine-tuned Large Language Model (Qwen3-8B) as the reasoning engine. It is integrated with a perception front-end (Vision Foundation Model) to generate scene graphs and an action expert back-end (meta-skill library based on geometric fabrics) to execute low-level control.
• Reasoning Process: The model reasons step-wise over object states and spatial relations to infer action preconditions, consequences, and goal satisfaction within a physically grounded space, rather than relying on implicit latent embeddings.

2. Manip-Cognition-1.6M Dataset

To support the learning of such structured reasoning, the authors constructed a large-scale dataset comprising 1.6 million samples, jointly supervising world understanding, action planning, and goal interpretation. The data engine processes raw visual demonstrations into structured task trajectories and includes three core components:

• World Understanding (15k samples): Trains the model to abstract visual observations into semantically grounded Scene Graphs (Text $\to$ SG pairs), bridging perception and symbolic reasoning.
• Action Planning (1.3M samples): Decomposes reasoning into three tasks:
  • Forward Action Reasoning: Inferring multi-step action sequences to bridge state transitions.
  • World Modeling: Predicting environmental consequences ($\Delta SG_{edge}$) of specific actions.
  • Goal-conditioned Planning: Predicting the next symbolic action given a current state and a textual goal.
• Goal Interpretation (288k samples): Focuses on "long-horizon goal dreaming," predicting the final environment configuration ($SG_{goal}$) given a current state and natural language instruction.

3. Training Strategy

The GSR model is trained in two stages:

• Stage 1 (Supervised Fine-Tuning - SFT): Uses Low-Rank Adaptation (LoRA) on the Qwen3-8B architecture to learn structured reasoning and action commands.
• Stage 2 (Reinforcement Fine-Tuning - RFT): Employs Group Relative Policy Optimization (GRPO) to align the model with physical execution constraints. The reward function targets three failure modes:
  • Step-wise Constraint Reward: Penalizes generating multiple actions in a single output, enforcing strictly step-by-step generation.
  • Scene-Graph Grounding Reward: Prevents object hallucinations by ensuring predicted object identifiers exist in the current scene graph.
  • Termination Reward: Differentiates between correct task completion and premature termination or infinite looping.

Key Contributions

1. GSR Framework: Introduction of a structured reasoning paradigm that explicitly models world-state evolution as transitions over semantically grounded scene graphs, separating high-level conceptual reasoning from low-level execution.
2. Manip-Cognition-1.6M: A large-scale dataset providing joint supervision for world understanding, intention interpretation, and action planning across diverse manipulation skills.
3. Explicit Inductive Bias: Demonstration that explicit world-state representations serve as a key inductive bias for scalable embodied reasoning, improving zero-shot generalization and long-horizon task completion.

Experimental Results

Extensive evaluations were conducted across RLBench, LIBERO, a custom GSR-benchmark, and real-world robotic tasks.

• Zero-Shot Generalization: On RLBench and LIBERO benchmarks, GSR (8B) significantly outperformed prompting-based baselines (including GPT-5, Gemini-2.5-Pro, and DeepSeek-V3) across various categories (Kitchen Operations, Pick and Place, Stacking, etc.). For instance, on RLBench Stacking and Assembly, GSR achieved 41.2% success compared to 15.0% for GPT-5.
• Long-Horizon Reasoning: On the GSR-benchmark (180 tasks, average horizon >10 steps), GSR achieved State-of-the-Art (SOTA) performance in Semantic Object Disambiguation, Spatial-Aware Sequencing, and Goal-conditioned Generalization.
• Robustness: GSR demonstrated robustness to perceptual noise. With 5% noise injected into scene graphs, Task Progress (TP) decreased only slightly, whereas goal-conditioned tasks showed higher sensitivity but remained stable compared to baselines.
• Real-World Deployment: The framework was successfully deployed on a Tianji Marvin-6 7-DoF dual-arm robot, demonstrating capabilities in pick-and-place, sorting, and daily-life tasks.

Significance and Claims

The paper claims that explicit world-state representations are a critical inductive bias for scalable embodied reasoning. By moving away from implicit latent representations that conflate task structure with perceptual variability, GSR enables agents to perform rational, step-wise reasoning about action preconditions and consequences. The authors posit that this approach allows for flexible composition of sequenced actions and robust adaptation across tasks, addressing the limitations of both end-to-end VLA models (which lack structural priors) and classical TAMP frameworks (which lack scalability to open-ended environments). The work highlights that grounding decision-making in explicit, structured representations supports more systematic generalization and sustained reasoning over time in open-world embodied settings.

Limitations

The authors acknowledge current limitations:

1. The framework relies on predefined action primitives, which may constrain flexible action generation and adaptation between successive steps in long-horizon tasks.
2. The vision representations used to construct scene graphs have limited ability to capture fine-grained geometry, subtle object states, and nuanced physical properties, potentially affecting accuracy in visually complex or cluttered environments.