💻 computer science

GraphPoint: Semantic Entity Graphs and Point Trajectories for Compositional Robot Manipulation

This paper introduces GraphPoint, a framework that links semantic entity graphs to geometric control via predicted gripper trajectories to improve instruction-dependent generalization in robot manipulation, validated by the new CoMani benchmark designed to test compositional reuse across subtasks and within subtasks.

Original authors: Kang Luo, Hesheng Wang

Published 2026-09-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kang Luo, Hesheng Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots that can pick up a cup or place a bowl on a table have become common sights in research labs, yet these machines often struggle when the instructions change slightly. A robot trained to put a bowl on a plate might fail if asked to put the same bowl to the right of the plate, or if asked to sweep it there instead of lifting it. This happens because many current systems learn to match a specific picture of a scene with a specific movement, rather than truly understanding the words describing what to do. When the visual scene looks familiar, the robot relies on that visual shortcut and ignores the new instruction. To build robots that can truly adapt, researchers need to teach them to separate the meaning of an object from its location and to understand how actions can be mixed and matched in new ways.

A team of researchers at Shanghai Jiao Tong University has developed a new approach to solve this problem, introducing a system called GraphPoint. Instead of treating the robot's view as a single, unstructured image or a cloud of points, this system breaks the scene down into a map of specific roles. It identifies the robot's hand, the object being moved, and the target destination as distinct entities. The system then uses a large language model to read the human instruction and translate it into a structured plan that defines exactly how these roles should interact. For example, if a person says "put the bowl on the plate," the system understands that the bowl is the object to be moved, the plate is the target, and the action is placing. Crucially, it keeps these roles separate in its internal logic, allowing it to apply the same "placing" logic to a different object or a different target without needing to relearn the entire movement from scratch.

The researchers tested their idea using a new set of challenges they created, which they named CoMani. This testing ground was designed to isolate specific types of learning. In one set of tests, the robot was asked to perform the same action, like putting an object down, but with different instructions about where to place it, such as "on the plate" versus "to the right of the plate." In another set, the robot had to use different types of movements, like sweeping an object instead of lifting it. Finally, they tested whether the robot could string together several simple tasks into a longer sequence it had never seen before, such as moving a bottle, then a bowl, then a piece of cheese, in a specific order. The goal was to see if the robot could rely on the words it heard rather than just memorizing the visual patterns of the room.

The results showed that GraphPoint significantly outperformed other leading methods in these tests. When the instruction changed the relationship between objects, such as asking the robot to place an item to the right instead of on top, GraphPoint succeeded in nearly all cases, while other systems often failed because they were stuck on the visual layout of the scene. The system also proved capable of learning new action types. When asked to rotate a knob, a task it had never been shown, it successfully applied the concept of rotation to a new object. Most impressively, when faced with a three-step sequence it had never practiced as a whole, the system was able to complete the first two steps correctly in over 80 percent of attempts and finish all three steps in about half of the trials. This demonstrated that the robot could take simple skills it had learned individually and combine them to follow complex, multi-step instructions.

The success of this system relies on how it processes information. Rather than working with absolute positions in the room, the system describes everything relative to the robot's own hand. This allows the robot to understand that moving an object "to the right" means the same thing regardless of where the object started. The system also uses a technique where it temporarily hides the detailed shape of objects during training, forcing it to pay attention to the language instructions and the roles of the objects instead of just memorizing what the objects look like. By grounding the robot's movements in a semantic map of roles and relationships, the researchers created a policy that responds to language changes even when the visual scene remains the same. This work suggests that for robots to become truly useful in dynamic environments, they must learn to treat instructions as the primary guide for action, rather than treating visual patterns as the only source of truth.

Technical Summary: GraphPoint

Problem Statement
Robot manipulation policies often fail to generalize beyond their specific training demonstrations, even when new instructions involve familiar objects and behaviors. When language and visual scenes are strongly correlated during training, policies tend to learn fixed visual–action mappings (shortcuts) rather than responding to the requested behavior. This limitation manifests in two forms of compositional generalization:

Within-subtask generalization: The ability to recombine familiar semantic factors (entities, action types, and modifiers) in new ways within a single atomic interaction.
Across-subtask generalization: The ability to reuse learned atomic subtasks to execute unseen long-horizon instructions without explicit sequence-level demonstrations.

Existing approaches, including image-based, scene-point, and entity-point policies, improve performance on demonstrated tasks but struggle when familiar entities are paired with new action modifiers or types, or when transitions between subtasks must be managed dynamically.

Methodology: GraphPoint
The authors propose GraphPoint, a framework that connects semantic entity graphs to geometric control. The core idea is to represent the current manipulation state as a semantic entity graph, making instructed relations explicit rather than relying on implicit visual correlations.

Task-to-Graph Construction: A Large Language Model (LLM) parses natural language instructions into ordered subtask graphs. Each subtask defines semantic roles: Actor (A), Patient (P), and Target (T). These roles are represented by learned embeddings. Action types and modifiers are encoded using a frozen language encoder (BGE) and projected to condition the policy.
Grounded Point Entities: Visual localization (using a VLM) and temporal tracking (using SAM 2) identify task-relevant objects. These are represented as sparse 3D point clouds (32 points per entity) relative to the current Tool Center Point (TCP). The actor (gripper) is represented by six geometric keypoints.
Role-Structured Entity Graph Encoder: The encoder processes entity points using a shared MLP and alternates between local encoding (within each entity) and global interaction (across entities).
- Relative Coordinates: All points are expressed relative to the TCP, reducing dependence on absolute scene placement.
- Conditioned Attention: The global attention mechanism exchanges information between entity summaries conditioned on action types and modifiers. A specific "patient-mediated" interaction chain ( $A \leftrightarrow P \leftrightarrow T$ ) is enforced, masking direct Actor–Target attention to encourage interaction through the manipulated object.
- Role-Aware Point Collapse (RPC): During training, point sets for patients or targets are randomly collapsed to their centroids to prevent the model from relying on shape-based shortcuts, forcing reliance on semantic conditions.
Point-Trajectory Flow Policy: The policy predicts future gripper point trajectories (10 steps) and gripper commands using conditional flow matching. The trajectory is predicted in the TCP-relative frame. Executable poses are recovered via least-squares rigid alignment of predicted keypoints to the robot's geometry.
Long-Horizon Composition: A progress head estimates subtask completion based on the relationship between the patient and target. This enables "self-switching" (SS), where the system advances to the next subtask based on predicted progress rather than external signals, allowing a single atomic policy to be reused across a sequence.

Benchmark: CoMani
To rigorously evaluate these capabilities, the authors introduce CoMani, a benchmark with controlled splits:

CoMani-Mod: Tests modifier generalization (e.g., "on" vs. "right of") with fixed action types. It uses matched initial scenes to ensure failures are due to relation understanding, not visual shortcuts.
CoMani-Act: Tests action-type generalization (e.g., "put" vs. "sweep" vs. "rotate") with fixed modifiers. It evaluates Action-Consistent Success Rate (ACSR) to ensure the robot performs the instructed action, not just reaches the goal.
CoMani-Seq: Tests sequential composition by chaining atomic subtasks into 3-step instructions without sequence-level training data.

Key Results
Experiments on CoMani demonstrate that GraphPoint outperforms diverse baselines (including ACT, Diffusion Policy, DP3, Point Policy, and CbF):

Modifier Generalization: GraphPoint achieves a mean Success Rate (SR) of 96.4% (80.0% on Out-of-Distribution tasks), significantly outperforming baselines that often fail to generalize to held-out modifiers.
Action Generalization: GraphPoint attains the highest Action-Consistent Success Rate (ACSR) on held-out actions, including transferring the "rotate" action to a new entity (cream cheese) where other methods failed completely.
Sequential Composition: GraphPoint is the only method capable of executing unseen 3-step sequences. Using environment-switching (ES), it achieves 82.5% SR for the first two subtasks and 50.8% for the full sequence. When using self-switching (SS), it achieves 95.0% SR2 and 67.5% SR3. Baselines rarely completed even the first two steps.
Ablation Studies: Removing Role-Aware Point Collapse (RPC) or using absolute coordinates drastically reduced OOD performance, confirming the necessity of semantic regularization and relative frames. Removing language conditioning reduced OOD success from 80% to 8.3%.

Significance and Claims
The paper claims that GraphPoint successfully addresses instruction-dependent generalization at both the atomic and sequential levels. By explicitly structuring the policy around semantic roles and conditioning interactions on language-derived modifiers and action types, the framework avoids the visual shortcuts that plague standard imitation learning. The use of a semantic entity graph allows the robot to interpret the same visual scene differently based on the instruction, and the predicted progress mechanism enables the composition of learned skills into complex, unseen tasks. The authors note that limitations currently lie in front-end perception (parsing, localization, segmentation) and severe occlusion, which can disrupt the tracked entity points.

Technical Summary: GraphPoint

More like this