💻 computer science

RelAfford6D: Relational 6D Affordance Graphs for Constraint-Driven Robotic Manipulation

RelAfford6D is a novel, training-free framework that bridges abstract semantics and precise physical control by constructing relational 6D affordance graphs to translate free-form instructions into kinematic constraint satisfaction problems, enabling robust zero-shot robotic manipulation of articulated objects through analytically formulated trajectory tracking.

Original authors: Guodong Zhang, Qichen He, Wenyuan Xie, Shaokai Wu, Yanbiao Ji, Qiuchang Li, Bayram Bayramli, Yue Ding, Hongtao Lu

Published 2026-06-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Guodong Zhang, Qichen He, Wenyuan Xie, Shaokai Wu, Yanbiao Ji, Qiuchang Li, Bayram Bayramli, Yue Ding, Hongtao Lu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to open a drawer, screw in a bottle cap, or close a laptop. The biggest challenge isn't just telling the robot what to do; it's helping the robot understand how the object physically moves.

Most current robots are like students who have memorized a specific dance routine. If you ask them to open a drawer, they know exactly where to grab it and how to pull. But if you give them a slightly different drawer, or if the drawer gets bumped while they are pulling, they often freeze or break the handle because they don't understand the rules of how that drawer is built. They rely on "guessing" based on patterns they saw during training.

RelAfford6D is a new way of thinking that stops the robot from guessing and starts it from understanding. Here is how it works, broken down into three simple steps:

1. The "Relationship Map" (Semantic Topology Generation)

Instead of just looking at an object as a single blob, this system asks the robot to think like a mechanic.

The Analogy: Imagine you are reading a recipe. A normal robot might just see "cake." This system sees "flour mixed with eggs, baked in a pan."
How it works: When you say, "Open the microwave," the system doesn't just look for a microwave. It instantly figures out the relationship between the parts. It identifies:
- The Handle: The part you touch.
- The Body: The part that stays still (the anchor).
- The Rule: "The handle rotates around the body."
It creates a "map" that says: "To open this, the handle must spin around the body, not slide away." It does this without needing to have seen that specific microwave before, using a built-in library of how things usually move.

2. The "3D GPS" (Metric Visual Grounding)

Once the robot knows the rules (the map), it needs to know exactly where the parts are in the real world.

The Analogy: Imagine you have a map of a city, but you are standing in a foggy street. You know you need to go to the "Library," but you don't know which building is the library.
How it works: The robot looks at the camera feed (RGB-D images). It uses advanced AI to find the exact 3D location and orientation of the "Handle" and the "Body." It turns the abstract idea of "handle" into a precise set of coordinates in space (like a 6D GPS lock). It knows exactly how the handle is tilted and how far it is from the body.

3. The "Train on Tracks" Execution (Constraint-Driven Kinematic Execution)

This is the most important part. Most robots try to fly freely through the air to get to a target. RelAfford6D puts the robot on tracks.

The Analogy: Imagine a train. A normal robot tries to drive a car to the station; if the road is blocked, it crashes. RelAfford6D is like a train on a track. The track is the physical rule (e.g., "rotate around this hinge"). The robot cannot leave the track.
How it works: The system calculates the exact mathematical path the robot's hand must follow to satisfy the physical rule.
- If it's a drawer, the track is a straight line (sliding).
- If it's a microwave, the track is a circle (rotating).
The Safety Net: If someone bumps the microwave while the robot is opening it, the robot doesn't panic. It has a "closed-loop" system (like a self-correcting GPS). It sees the microwave moved, instantly recalculates the "track" based on the new position, and adjusts its path on the fly to finish the job.

Why is this a big deal?

The paper claims that because this system relies on physics rules rather than memorized patterns, it is incredibly robust.

Zero-Shot: It can handle objects it has never seen before because it understands the logic of how they move, not just what they look like.
No Training: You don't need to spend weeks teaching the robot with thousands of videos. It just needs to understand the language and the physics.
Resilience: If the world changes (someone moves the object), the robot adapts instantly because it is constantly checking the physical "tracks" rather than following a pre-recorded script.

In short, RelAfford6D turns a robot from a "parrot" that repeats what it saw into a "mechanic" that understands how things work, allowing it to manipulate the open world with precision and confidence.

Technical Summary: RelAfford6D

Problem Statement

Open-world robotic manipulation faces a fundamental "perception-action gap": the difficulty of transforming high-dimensional, multi-modal observations into precise, low-level physical control commands. While recent data-driven policies and foundation model-based paradigms have attempted to bridge this gap, they suffer from a critical bottleneck in object representation. Existing approaches often rely on isolated contact points (e.g., 2D heatmaps, sparse 3D keypoints) or implicit energy fields. These representations lack the rigorous, explicit kinematic constraints necessary for manipulating complex articulated objects. For instance, successfully opening a microwave is not merely about reaching a handle; it is strictly governed by the rigid-body relative geometry between the door and the main body. Current methods frequently fail to encode these continuous relative geometries, leading to kinematic violations (e.g., pulling a revolute door in a straight line) and poor generalization in open-world settings without task-specific training.

Methodology

The authors propose RelAfford6D, a novel, training-free framework centered on a Relational 6D Affordance Graph. This framework bridges abstract semantic reasoning and physical control through a three-stage pipeline:

1. Semantic Topology Generation

This module addresses the challenge of deducing the underlying kinematic chain of unseen objects from free-form natural language instructions.

Knowledge Base: The system utilizes a structured kinematic knowledge base derived from the PartNet-Mobility dataset. Objects are abstracted into relational pairs consisting of a primary interacting part ( $s_p$ ) and a physical anchor part ( $s_a$ ), linked by a kinematic constraint type (e.g., revolute, prismatic).
Retrieval-Augmented Reasoning: A Large Language Model (LLM) is guided by Retrieval-Augmented Generation (RAG). The system retrieves relevant kinematic chains from the knowledge base based on the user query and injects them as priors into the LLM.
Output: The LLM outputs a rigorous relational node topology $((s_p, s_a), a)$ , explicitly identifying the interacting parts and the specific geometric constraint manifold (e.g., "rotate" implies a revolute constraint).

2. Metric Visual Grounding

This module elevates the abstract topological nodes into precise metric SE(3) poses, bridging the semantic-geometric gap.

Segmentation: Using the SAM3 vision foundation model, the system performs language-guided segmentation to generate fine-grained binary masks for the primary and anchor parts based on the inferred topology.
Pose Estimation: The masked RGB-D regions are processed by a category-agnostic pose estimator (FoundationPose). This step instantiates the abstract nodes into explicit 6D poses ( $T^*_P$ and $T^*_A$ ) in the world coordinate system.
Result: The output is an instantiated spatial graph $\Psi = \{(T^*_P, T^*_A)\}$ , providing the exact boundary conditions required for physical execution without relying on pre-trained category-specific models.

3. Constraint-Driven Kinematic Execution

The final module formulates action execution as a continuous SE(3) kinematic constraint satisfaction problem, replacing learned policies with geometric feedback control.

Manifold Definition: The semantic token (e.g., "pull," "rotate") acts as a selector for a specific geometric constraint manifold ( $M_a$ $M_{a}$ ).
- Prismatic (Pull): Motion is restricted to a 1D translation along an axis defined relative to the anchor.
- Revolute (Rotate): Motion follows a 1D rotational orbit around a hinge axis anchored at the anchor part.
Trajectory Synthesis: The system analytically synthesizes trajectories by tracking these manifolds. A global planner (RRT-Connect) handles obstacle avoidance for the reaching phase, while the local execution phase strictly follows the 1D constraint manifold.
Closed-Loop Replanning: To ensure robustness against dynamic disturbances, the system employs a closed-loop tracking mechanism (5 Hz) using FoundationPose. If the tracked anchor pose deviates from the planned state beyond a geometric threshold, the system immediately re-instantiates the manifold using live coordinates and analytically re-synthesizes the remaining trajectory.

Key Contributions

Relational 6D Affordance Graph: A novel intermediate representation that models manipulation as part-conditioned SE(3) relations, explicitly encoding rigid-body and articulation constraints rather than isolated contact points.
Structured Part-Level Kinematic Ontology: A method for encoding reusable relational priors that enables zero-shot semantic topology inference directly from natural language, eliminating the need for category-specific templates.
Closed-Loop Constraint Optimization: A reformulation of manipulation as a continuous optimization on a relational SE(3) constraint manifold. This replaces learned end-to-end policies with physically grounded geometric feedback control.
Training-Free Generalization: The framework achieves robust zero-shot performance in both simulation and real-world environments without requiring any task-specific training or demonstrations.

Experimental Results

The authors evaluated RelAfford6D against state-of-the-art baselines (including Where2act, FlowBot3D, ManipLLM, and RoboMamba) in two settings:

Articulated Object Manipulation: On a benchmark of 10 complex kinematic tasks (e.g., opening drawers, screwing bottles), the training-free RelAfford6D achieved an average success rate of 74%. This outperformed all data-driven baselines, which required training on 10,000 trajectories. The authors attribute this to the explicit enforcement of SE(3) manifold constraints, which prevents kinematic violations common in methods that infer isolated contact points.
Rigid Object Manipulation: On classical tasks (pick, stack, push cubes), the method achieved 100% success rates, demonstrating that the framework generalizes beyond articulated mechanisms to rigid-body tasks without re-training.
Ablation Studies: Experiments confirmed that removing kinematic knowledge retrieval, replacing vision foundation models, or disabling dynamic manifold replanning significantly degraded performance (dropping success rates to between 35% and 68.5%), validating the necessity of each component.

Significance and Claims

The paper claims that RelAfford6D fundamentally addresses the tension between expressiveness and kinematic physical constraints in robotic manipulation. By shifting the paradigm from "learning to act" via latent features to "solving to act" via explicit geometric constraints, the system achieves:

Robustness: Unprecedented resistance to dynamic external disturbances through continuous closed-loop replanning.
Generalization: Superior zero-shot capabilities across diverse object categories and task types without task-specific data.
Physical Validity: The generation of kinematically valid trajectories that strictly adhere to the physical joint geometry of articulated objects, a capability often missing in data-driven approaches.

The authors position this work as a step toward enabling robots to reliably interact with physically diverse, unseen objects in open-world scenarios by grounding abstract semantics in rigorous physical control.