💻 computer science

AffordTrajDP: Dynamic Affordance-Guided Visuomotor Policy Learning for Robotic Manipulation

AffordTrajDP is a dynamic framework that enhances robotic manipulation by propagating a retrieved anchor affordance through an object's SE(3) pose to generate temporally consistent, state-aware guidance trajectories, thereby overcoming the limitations of static affordances and achieving superior success rates in precision-critical tasks.

Original authors: Gaoyuan Wu, Ziyu Shan, Haoyang Du, Yuyao Jiang, Ziwei Wang

Published 2026-08-04

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Gaoyuan Wu, Ziyu Shan, Haoyang Du, Yuyao Jiang, Ziwei Wang

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine teaching a robot to do chores. For a long time, scientists tried to teach robots by showing them videos of humans doing tasks, hoping the robot would just "watch and learn." But robots are notoriously bad at this when things change. If you move a cup three inches to the left, or if the lighting changes, a robot trained on the old setup might get confused and drop the cup. It's like trying to follow a recipe written for a specific kitchen, but then you move the stove to a different room; the instructions no longer make sense.

To fix this, researchers developed a clever trick called "affordance." Think of an affordance as a "magic highlight" on an object that tells the robot exactly where to grab or touch. Instead of looking at the whole messy room, the robot just looks at the highlighted spot. However, there's a catch: most of these highlights are static. They are like a sticker you put on a moving car. If the car drives forward, the sticker stays in the same spot in the air, and the robot, trying to touch the sticker, ends up missing the car entirely. This paper tackles that exact problem: how do you make the "magic highlight" move along with the object so the robot never loses its grip?

The researchers behind AffordTrajDP (a mouthful of a name, but think of it as "Affordance Trajectory Dynamic Policy") realized that the old way of using static highlights was the main reason robots failed at tricky tasks like inserting a USB drive or stacking blocks. Their solution is to stop treating the highlight as a fixed sticker and start treating it as a moving target.

Here is how their new system works, using a simple analogy: Imagine you are trying to catch a Frisbee thrown by a friend.

The Old Way (Static Affordance): Your friend tells you, "The Frisbee will be at the big oak tree." You run to the tree and wait. But your friend throws the Frisbee past the tree. You stand there, waiting for a Frisbee that never arrives at the tree, while the real Frisbee flies by. This is what happens when a robot uses a static contact point; it calculates where to touch based on where the object was, not where it is going.
The New Way (AffordTrajDP): Instead of a single spot, your friend gives you a moving path. "The Frisbee will be here, then here, then here." You don't just run to one spot; you run along a path that matches the Frisbee's flight.

In the robot's world, the "path" is called a dynamic affordance trajectory. The system first finds a "source" example from a memory bank—like looking at a photo of a cup to find the perfect spot to grab its handle. Then, instead of just copying that spot, the robot predicts where the object will be in the next few seconds. It takes that perfect grab-spot and "propagates" it forward, dragging it along with the object's predicted movement. This creates a moving guide that updates every millisecond, ensuring the robot's hand is always aligned with the object, even if the object wobbles or shifts.

The team tested this idea in two ways: in a super-accurate computer simulation called ManiSkill3 and in the real world using actual robot arms (a Galaxea A1 and a UR7e).

In the simulation, they challenged the robot with six different tasks, from picking up cubes to plugging in chargers. The results were clear: the old methods (like DP3 and AffordDP) managed to succeed about 52.2% to 57.8% of the time. But with the new dynamic guide, the success rate jumped to 70.0%. The improvement was most dramatic in the hardest tasks, like inserting a charger plug, where the old methods struggled to keep the plug aligned once it touched the socket.

They didn't stop at simulations. They took the robot to a real lab to try tasks like stacking cubes, picking up cups, and the notoriously difficult Adapter Insertion (fitting a specific plug into a tight socket). They tested the robot with objects it had seen before and objects it had never seen (different shapes or colors).

On the Galaxea A1 robot, the new method achieved a 66.1% success rate across all tasks, while the best previous method only managed 35.0%.
On the UR7e robot, the new method hit 72.5% success, beating the previous best of 55.8%.

The researchers also ran a "detective" experiment to see which part of their system was actually doing the heavy lifting. They tried removing pieces of the puzzle:

Just the path, no specific spot: If they told the robot to follow a path but didn't tell it where on the object to start, the robot got confused and failed (sometimes doing worse than the old methods).
Just the spot, no path: If they gave the robot the perfect grab spot but didn't update it as the object moved, the robot drifted off course after contact, especially on tricky tasks.
Both together: Only when they combined the specific "where to touch" with the "how it moves" did the robot succeed consistently.

This suggests that for robots to handle precision tasks in a messy, changing world, they need more than just a snapshot of where to grab; they need a moving roadmap that updates in real-time. The authors note that while their method is a significant step forward, it still faces challenges with extremely tight tolerances (like the USB insertion task, which was the hardest for everyone). However, by proving that dynamic, moving guides work better than static ones, they've opened a new door for making robots that are less likely to drop the ball when things don't go exactly as planned.

Technical Summary: AffordTrajDP

1. Problem Statement

Existing visuomotor policies for robotic manipulation, particularly those based on diffusion models, often rely on dense visual inputs (e.g., raw RGB or point clouds). While effective for generalization, these approaches are sensitive to environmental perturbations such as object appearance changes, background clutter, and pose shifts.

A specific limitation arises in precision-critical tasks (e.g., adapter insertion, peg-in-hole) where static affordance priors fail. Current state-of-the-art methods like AffordDP retrieve a static contact point and a post-contact trajectory from a reference demonstration, transferring them to the target scene via a single rigid transformation. However, once transferred, these priors remain fixed. If the object moves or shifts after the initial contact (due to placement perturbations or contact-induced deviations), the static prior becomes misaligned with the object's evolving pose. This leads to trajectory drift, oscillation, and failure in tasks requiring sustained, precise alignment.

The core problem identified is the absence of a conditioning signal that jointly preserves both:

Where to contact (semantic contact location).
How that contact location should evolve as the object moves (temporal consistency).

2. Methodology: AffordTrajDP

The authors propose AffordTrajDP, a dynamic framework that constructs affordance trajectories via object-centric temporal propagation. Instead of a static prior, the system generates a time-varying guidance signal that updates throughout the execution.

Core Pipeline

Anchor Affordance Retrieval (Semantic Localization):
- Given an RGB-D observation, the system retrieves the most similar source object instance from an object memory (built using GroundedSAM for cropping and CLIP for appearance embedding).
- It transfers the source's static contact point ( $c$ ) to the target object using dense visual correspondence. This provides the "where" (the anchor point).
Future Pose Estimation (Temporal Prediction):
- A diffusion-based trajectory model predicts the future SE(3) pose trajectory of the target object relative to its current frame.
- Unlike methods that condition directly on raw sensory inputs, this model predicts the object's motion trajectory ( $T_{tgt}(t)$ ) based on pose history, normalizing away absolute scene layout to improve cross-instance generalization.
- The model uses Denoising Diffusion Implicit Models (DDIM) to ensure deterministic sampling and reduce inference latency.
Dynamic Affordance Propagation:
- The retrieved static anchor contact point ( $c_{tgt}$ ) is propagated forward through the predicted future pose trajectory.
- Mathematically, for each future step $k$ , the contact point is updated via rigid transformation: $p_{t+k} = T_{t+k} \cdot \tilde{c}$ .
- This yields a dynamic affordance trajectory ( $\tau$ ) that continuously adapts to the object's predicted motion, providing state-consistent guidance for the post-contact phase.
Action Generation:
- The dynamic affordance trajectory is encoded (via a Transformer) and combined with the static affordance and other observations (point clouds, proprioception) to condition a diffusion policy.
- The policy generates actions by denoising from Gaussian noise, conditioned on this time-varying geometric prior.

3. Key Contributions

The paper outlines three primary contributions:

New Problem Formulation: The authors formalize the limitation of existing affordance-guided policies: static priors cannot maintain alignment with moving objects after contact. They propose AffordTrajDP as the first framework to construct a state-consistent, time-varying affordance trajectory to address this.
Necessity and Sufficiency of Joint Conditioning: Through controlled ablation studies, the authors demonstrate that semantic contact retrieval and pose-trajectory propagation are both necessary and sufficient for high performance.
- Contact retrieval alone improves performance but plateaus.
- Trajectory propagation without a semantic anchor is unreliable and can underperform baselines on unseen instances (e.g., dropping to 20% success on StackCube unseen).
- Only the combination yields consistent, large gains (66.1% vs. 35.0% baseline).
Empirical Validation: Extensive evaluation across simulation (ManiSkill3) and two real-world robot platforms (Galaxea A1 and UR7e) across seven tasks, including precision insertion and object alignment.

4. Experimental Results

Simulation (ManiSkill3)

Evaluated on six tabletop tasks (PickCube, PegInsertion, PlugCharger, StackCube, PokeCube, PushCube) with a Franka Panda robot.

Overall Success Rate: AffordTrajDP achieved 70.0%, outperforming DP3 (52.2%), AffordDP (57.8%), and SPOT (62.1%).
Precision Tasks: The largest gains were observed in precision-critical insertion tasks (PegInsertion, PlugCharger), where dynamic guidance significantly reduced drift compared to static baselines.

Real-World Experiments

Evaluated on two platforms:

Galaxea A1 (L515 camera): Tasks included StackCube, PickCup, and AdapterInsertion.
UR7e (D455 camera): Tasks included PickCup, Ring-on-Peg, Put-in-Bowl, and USB Insertion.
Generalization: The method was tested on both seen and unseen object instances (varying in appearance and geometry).
Performance:
- On Galaxea A1, the full method achieved 66.1% average success rate, significantly outperforming the DP3 baseline (35.0%).
- On UR7e, it achieved 72.5% average success rate, outperforming DP3 (40.8%) and SPOT (54.2%).
- AdapterInsertion: The most significant improvement was seen here, where static priors often failed due to post-contact drift, while AffordTrajDP provided stabilizing guidance.

Ablation Insights

Semantic Anchor: Using a semantic contact point is superior to using the object center, especially for unseen instances.
Trajectory Propagation: Propagating the contact point along the predicted trajectory is critical. Without the semantic anchor, trajectory conditioning acts as noise.
Error Analysis: The authors observed that pose-trajectory prediction error does not compound into growing drift over the episode duration in their specific test conditions, suggesting the propagation mechanism remains robust.

5. Significance and Claims

The paper claims that AffordTrajDP addresses a fundamental gap in affordance-guided imitation learning: the disconnect between where to interact and how that interaction evolves over time.

Robustness: By providing state-consistent, time-varying guidance, the method stabilizes post-contact motion, making it particularly effective for precision-critical tasks where small misalignments accumulate.
Generalization: The approach demonstrates strong generalization to unseen object instances and diverse robot embodiments, outperforming methods that rely solely on visual generalization or static priors.
Modest Claims: The authors are modest regarding the error analysis. They acknowledge that their finding (that prediction error does not compound) is based on a correlational, single-task comparison rather than a controlled ablation with injected noise. They frame the hypothesis that this non-accumulation property holds under heavy occlusion or noise as a direction for future work, rather than an established result.
Future Directions: The paper suggests extending this framework to include richer contact semantics, uncertainty-aware pose prediction, and tactile/force feedback to handle a wider range of manipulation tasks.