🤖 machine learning

Semantic-Geometric Task Representations for Bimanual Manipulation from Human Demonstrations to Robot Action Planning

This paper introduces a semantic-geometric graph-based representation that decouples task-agnostic encoding from action-conditioned decoding to enable robust bimanual manipulation planning across varying task structures and robot embodiments, achieving superior performance on real-robot tasks compared to existing baselines.

Original authors: Franziska Herbert, Vignesh Prasad, Han Liu, Dorothea Koert, Georgia Chalvatzaki

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Franziska Herbert, Vignesh Prasad, Han Liu, Dorothea Koert, Georgia Chalvatzaki

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to cook dinner or clean up a messy table using two hands. If you just show the robot a video of you doing it, the robot might get confused. It might think, "Okay, I need to pick up the spoon," but then it forgets why it's picking it up, or it might grab the bowl instead of the spoon because it's focusing only on the shapes it sees, not the story of what's happening.

This paper introduces a new way to teach robots by giving them a "mental map" that combines what things are (semantics) with where things are and how they move (geometry).

Here is a simple breakdown of how it works:

1. The Problem: Robots Need a Story, Not Just a List

When humans do tasks with two hands, the order of actions can change. Sometimes you stir the soup before you pour it; sometimes you pour then stir. Sometimes you use your left hand for the bowl and right for the spoon; other times it's swapped.

Old methods were like reading a grocery list: "Buy milk, buy eggs." They didn't understand the relationship between the items or the flow of the story.
This paper's method is like reading a comic book. It understands that "The spoon is inside the bowl" (a relationship) and "The spoon is moving up to stir" (geometry).

2. The Solution: The "Graph" (A Web of Connections)

The researchers built a system that turns the scene into a Graph. Think of a graph like a subway map:

The Stations (Nodes): These are the objects (the bowl, the whisk, the left hand, the right hand).
The Tracks (Edges): These are the connections between them. The tracks tell the robot if the whisk is touching the bowl, or if the hand is approaching the bottle.
The Timetable (Time): The system doesn't just look at one snapshot; it watches the whole movie, remembering where things were a second ago and where they are going next.

3. The Brain: Two Parts Working Together

The AI model has two main parts, like a Director and a Scriptwriter:

The Encoder (The Director): This part watches the human demonstration and builds the "mental map" (the graph). It learns the structure of the task. Crucially, it learns this structure without memorizing specific robot movements. It's like a director who understands the plot of a play but hasn't decided which actor will play the role yet.
The Decoder (The Scriptwriter): This part takes the Director's map and writes the specific instructions for the robot. It predicts: "Next, the robot should grab the whisk," and "The whisk should move in this specific arc."

The Magic Trick: Because the Director (Encoder) learns the general story of the task, you can use the same Director for different robots. You just swap out the Scriptwriter (Decoder) and give it a few examples of how this specific robot moves, and it learns to adapt instantly.

4. The Test: From Human to Robot

The researchers tested this on 11 different tasks, like cooking (stirring, pouring) and cleaning (clearing a table).

The Results: When they tried to teach a real, physical two-handed robot, their "Graph" method worked perfectly. It successfully completed the tasks.
The Competition: They compared it to other AI models (like standard Transformers or models that only look at sequences).
- The other models got stuck when the tasks got messy or the order of actions changed.
- The "Graph" model was the only one that could handle the chaos of a messy table and still figure out the right steps.
- Even a very advanced "Vision-Language" model (which is good at reading text and seeing images) failed completely on the robot, getting confused about which hand should do what.

5. The "Safety Check"

Before the robot actually moves, the system does a quick "reality check." It predicts the path the object will take and asks: "Does this path make sense with the robot's physical limits?" If the robot tries to reach across the table in an impossible way, the system says, "No, that won't work," and picks a different plan. This prevents the robot from crashing or dropping things.

Summary

In short, this paper teaches robots to understand tasks not just as a list of steps, but as a dynamic story of objects interacting. By using a "graph" to track relationships and movement, the robot can learn from a human's messy, variable demonstrations and then successfully perform those same tasks on a real machine, even when the situation changes. It's the difference between a robot that blindly follows a script and a robot that actually understands what it's doing.

Technical Summary: Semantic–Geometric Task Representations for Bimanual Manipulation

Problem Statement
Bimanual manipulation tasks exhibit significant variability in execution, including the ordering of actions, the specific objects involved, and the geometric nature of interactions. Learning generalizable task representations from human demonstrations requires capturing both the discrete semantic structure of the task (what is happening) and the continuous geometric evolution of the scene (how it happens). Existing approaches often fail to integrate these two modalities; graph-based methods typically focus on either semantic relations or geometric trajectories but rarely unify them to drive online robotic decision-making. Furthermore, many current methods are limited to frame-wise action recognition or single-step predictions, lacking the capacity to forecast long-horizon task progression. The challenge lies in creating a representation that jointly encodes object identities, inter-object semantic relations, and per-object motion histories in a form that supports reasoning over task progression and transfers across different embodiments (e.g., from human to robot).

Methodology
The authors propose a semantic–geometric task graph-representation learned via a decoupled encoder-decoder architecture.

Graph Structure: Each manipulation demonstration is modeled as a spatio-temporal graph $G(t) = (V, E, u)$ .
- Nodes: Represent objects and hands. Unlike prior work that creates separate nodes for each time step, this approach uses a single node per object with temporal features (concatenated object IDs and 3D coordinates from $H$ past frames) to reduce complexity.
- Edges: Capture semantic relationships (static spatial relations like "right of" and dynamic relations like "approaching") between nodes.
- Global Features: Encode the task ID to provide global context.
- Input: The graph is constructed from RGB-D video data, with semantic relations computed heuristically from object positions.
Encoder (MPNN): A Message Passing Neural Network (MPNN) processes the graph. It iteratively updates node, edge, and global embeddings over $K$ iterations. Crucially, the encoder operates solely on the temporal scene graph to produce structured representations decoupled from specific action labels. It utilizes Rotary Position Embeddings (RoPE) to handle temporal dependencies and alternates message passing with temporal self-attention.
Decoder (Transformer-based): The decoder conditions on the learned graph embeddings to forecast future task progression. It employs a multi-stage design to handle different time scales:
- Stage 1: Predicts the immediate next action-object pair ( $t+1$ ).
- Stage 2: Predicts future high-level semantic action-object pairs ( $a^F, o^F$ ) to reason about task sequencing independently of current action duration.
- Stage 3: Predicts full action, object, and 3D motion trajectories over a prediction horizon $P$ .
- The architecture uses separate Transformer decoders for action/object sequences (attending to global embeddings) and motion sequences (attending to node embeddings).
Training and Planning:
- Loss: A joint loss function combines weighted cross-entropy for action/object classification and Mean Squared Error (MSE) for motion regression.
- Inference: To handle overlapping predictions, the system uses action chunking and temporal ensembles with exponential decay weighting.
- Robot Deployment: An online planner couples the decoder's semantic predictions with geometric motion predictions. It scores candidate actions by combining action-object probabilities with a trajectory consistency score derived from learned Probabilistic Movement Primitives (ProMPs). A symbolic precondition mask filters infeasible actions, and a feasibility check ensures workspace constraints are met.

Key Contributions

Semantic–Geometric Graph Representation: A unified graph structure that jointly encodes object identities, inter-object semantic relations, and per-object motion histories for bimanual manipulation.
Decoupled Architecture: An encoder-decoder design that separates scene representation learning from action-conditioned forecasting. This allows the encoder to be reused across different embodiments (human to robot) via decoder-only fine-tuning on small robot datasets.
Empirical Characterization: An analysis showing that the benefits of structured relational inductive biases grow with task variability in action ordering and object involvement.
Real-Robot Validation: A system-level demonstration where the learned representations drive online action planning on a physical bimanual robot, outperforming various baselines.

Results
The approach was evaluated on eleven bimanual tasks across two datasets (KIT Bimanual Actions Dataset and a custom dataset, Ours(Bimacs)).

Human Demonstration Prediction: The MPNN encoder consistently outperformed ablations (Dreher [8], Lagamtzis [6]), Transformers, and Decoder-only models, particularly in tasks with high action or object variability. It achieved superior accuracy in motion forecasting and object prediction, demonstrating the value of integrating semantic and geometric information.
Transfer to Robot: When transferred to a real bimanual robot, models pre-trained on human data and fine-tuned on a small set of robot demonstrations (10–15 per task) achieved significantly higher accuracy and lower motion RMSE than models trained solely on robot data.
Online Planning: In real-robot experiments on "Cooking" and "Clear Up" tasks:
- The proposed method achieved 100% task success on both tasks.
- It achieved the lowest motion prediction RMSE and the lowest planner infeasibility rate.
- Baselines, including a fine-tuned Vision-Language Model (VLM), failed significantly. The VLM achieved 0% success due to an inability to switch action predictions dynamically and high motion prediction errors.
- The "Clear Up" task, characterized by high object variability, highlighted the necessity of the structured representation, as simpler baselines failed to achieve full success.

Significance and Claims
The paper claims that learning task representations for bimanual manipulation requires the joint modeling of semantic relations and geometric evolution, extending beyond single-frame predictions. The significance of this work lies in demonstrating that:

Structured semantic–geometric representations are essential for generalizing across tasks with high variability in action ordering and object involvement.
A decoupled architecture enables the transfer of task knowledge from human demonstrations to physical robots with minimal robot-specific data (decoder-only fine-tuning).
Coupling semantic action predictions with geometric motion consistency checks (via ProMPs) is critical for robust online action planning, allowing the system to reject semantically plausible but geometrically infeasible actions.

The authors conclude that their approach provides a data-efficient substrate for bimanual manipulation that bridges the gap between human demonstration and robotic execution, offering a structured alternative to large-scale pretraining or purely sequence-based models.