DeLTa: Demonstration and Language-Guided Novel Transparent Object Manipulation

The paper proposes DeLTa, a novel framework that integrates depth estimation, 6D pose estimation, and vision-language planning to enable precise, long-horizon manipulation of novel transparent objects using only a single demonstration and natural language instructions.

The Gist

Imagine you are trying to teach a robot to do chores in your kitchen. Now, imagine those chores involve handling glass cups, clear plastic bottles, and jars of water.

For a human, this is easy. We see the glass, we know where it is, and we pick it up. But for a robot? It's a nightmare. Standard robot cameras are like wearing sunglasses that don't work on glass; they see right through the object or get confused by the reflections, thinking the glass isn't there at all.

This paper introduces DeLTa, a new "brain" for robots that solves this problem. Think of DeLTa as a super-smart, patient teacher who can watch you do a task once, understand your instructions, and then teach a robot how to do it perfectly—even if the robot has never seen that specific object before.

Here is how DeLTa works, broken down into simple concepts:

1. The "X-Ray Vision" Glasses (Seeing the Invisible)

Robots usually fail with transparent objects because their depth sensors (which tell them how far away things are) get tricked by light bending through glass.

• The Analogy: Imagine trying to measure the depth of a swimming pool using a ruler that only works on solid ground. It fails.
• The DeLTa Fix: DeLTa uses a special "AI-powered X-ray vision." Instead of trusting the raw camera data, it uses a smart algorithm to reconstruct the true shape and position of the glass, filling in the missing gaps so the robot knows exactly where the object is in 3D space.

2. The "One-Time Lesson" (Learning from a Single Video)

Usually, to teach a robot to pick up a specific cup, you might need to record the robot doing it 100 times with 100 different cups. That's slow and expensive.

• The Analogy: Imagine you want to teach a child how to pour milk. You don't need to show them how to pour milk from a cow, a goat, and a sheep. You show them once with a cow, and they figure out the motion applies to any container.
• The DeLTa Fix: DeLTa only needs one video of a human doing a task (like picking up a bottle or pouring water). It extracts the "soul" of the movement—the path the hand took—and then mathematically "morphs" that path to fit a new, different transparent object. It's like taking a dance routine and teaching it to a new dancer with a different body size; the steps remain the same, but the scale adjusts automatically.

3. The "Project Manager" (Understanding Your Words)

You can tell DeLTa, "Can you make a green liquid in the cylinder?" or "Put the bottles in a straight row on the shelf."

• The Analogy: If you ask a regular robot, "Put the bottle on the shelf," it might try to grab the bottle while its arm is still in the way of the shelf, causing a crash. It lacks common sense.
• The DeLTa Fix: DeLTa uses a Vision-Language Model (VLM), which is like a project manager. It breaks your big request into tiny, logical steps.
  • Step 1: "Look for the bottle."
  • Step 2: "Move the arm out of the way."
  • Step 3: "Grab the bottle."
  • Step 4: "Check if the shelf is clear."
  • Step 5: "Place it."
    If the robot tries to grab the bottle before looking for it, the "Project Manager" catches the mistake, says "Wait, you can't grab what you haven't found yet!", and rewrites the plan.

4. The "Last Inch" Precision (The Final Touch)

Getting the robot's arm to the general area is easy. Getting it to precisely pour liquid into a small glass without spilling is hard.

• The Analogy: It's like driving a car to a parking lot (easy) versus parallel parking a long truck in a tiny spot (hard).
• The DeLTa Fix: Once the robot is close, DeLTa switches to a "Last-Inch Planner." It uses the 3D map it built earlier to navigate the final few inches with extreme care, avoiding collisions with the table or other objects, ensuring the liquid goes exactly where it should.

Why This Matters

Before this, robots were great at moving boxes but terrible at handling delicate, see-through things like glassware or chemicals.

• Old Way: "I can only move this specific red box I was trained on."
• DeLTa Way: "You want me to organize these weird glass jars? Sure. I watched you do it once, I understand your words, and I can figure out the geometry of these new jars instantly."

In short: DeLTa gives robots the ability to see through the invisible, learn from a single glance, and think through complex instructions, making them ready for real-world jobs in kitchens, labs, and stores where glass and clear plastic are everywhere.

Technical Summary

1. Problem Statement

Robotic manipulation of transparent objects (e.g., glassware, bottles, lab equipment) remains a significant challenge in real-world environments due to two primary limitations:

• Perception Failure: Standard depth sensors (like RGB-D cameras) fail to capture accurate geometry for transparent objects because light refracts or reflects off their surfaces, leading to erroneous or missing depth data.
• Generalization & Long-Horizon Constraints: Existing methods are largely restricted to:
  • Short-horizon tasks: Primarily simple pick-and-place operations.
  • Category-level priors: Methods that generalize only within a specific object category (e.g., "bottles") but fail on novel instances or irregular shapes.
  • Lack of Precision: They struggle with tasks requiring precise 6D pose estimation (position + orientation) needed for alignment, pouring, or tight shelf stocking.
  • Limited Instruction Following: Few frameworks can execute complex, multi-step tasks driven by natural language instructions.

The paper addresses the need for a system that can perform precise, long-horizon manipulation of novel transparent objects using single human demonstrations and natural language instructions.

---

2. Methodology: The DeLTa Framework

DeLTa (Demonstration and Language-Guided Novel Transparent Object Manipulation) is a unified framework integrating perception, planning, and control. It operates in three main stages:

A. Parsing Human Demonstrations (Perception & Trajectory Extraction)

Instead of requiring robot demonstrations for every new object, DeLTa extracts trajectories from a single human video demonstration per skill primitive (Pick, Place, Pour).

• Transparent Depth Estimation: Uses FoundationStereo (a foundation model) to reconstruct metric-scale depth from stereo images, overcoming the limitations of raw ZED camera sensors which fail on transparent surfaces.
• Segmentation & Pose Estimation:
  • Uses open-vocabulary detection and segmentation to isolate hands and objects.
  • Estimates 6D object poses and hand poses (21 keypoints) using the reconstructed depth and 3D meshes.
  • Constructs a canonicalized coordinate system based on hand landmarks to define the trajectory relative to the object.
• Trajectory Database: The extracted Cartesian-space trajectories are subsampled, smoothed, and stored. Crucially, these trajectories are canonicalized (aligned to the object's frame), allowing them to be retargeted to novel objects later.

B. Vision-Language Guided Task Planning

A VLM-guided planner decomposes natural language instructions into executable robot actions.

• VLM Planner: A foundation Vision-Language Model (without fine-tuning) generates a high-level "skeleton plan" (e.g., Pick -> Pour -> Place) based on the task description and visual input.
• Plan Translation: Converts the VLM output into PDDL (Planning Domain Definition Language) to ensure syntactic correctness and define preconditions/effects.
• Grounded Plan Search (Refinement): This is a critical innovation. The VLM often ignores robot constraints (e.g., single-arm limitations, eye-in-hand camera field of view). DeLTa employs an iterative search-and-refinement algorithm:
  • It validates preconditions (e.g., "Is the object visible? Is the gripper empty?").
  • If a plan fails (e.g., missing a "LookFor" step), it triggers a backward breadth-first search to insert necessary intermediate actions (like moving the camera or placing an object down) to make the plan executable.
  • This bridges the gap between symbolic planning and physical execution constraints.

C. Demonstration-Guided Robot Action Execution

The robot executes the refined plan by adapting the stored human trajectories to the current novel object.

• Pose-Based Trajectory Retargeting:
  • The system estimates the 6D pose of the novel transparent object in the scene.
  • It applies a rotation-based alignment to the canonicalized trajectory. The trajectory is rotated around the target position so that the start point aligns with the robot's current end-effector pose, while the end point remains fixed on the target object's manipulation pose.
  • This allows a single "Pick" demonstration to be reused for any object instance without re-training.
• Last-Inch Motion Planner:
  • Global Planning: Generates a collision-free path to the start of the retargeted trajectory using a point cloud derived from the enhanced depth estimation.
  • Local Tracking: Uses a Quadratic Programming (QP) based Inverse Kinematics (IK) solver with adaptive accuracy. It prioritizes collision avoidance early in the motion and tightens precision as the end-effector approaches the target.

---

3. Key Contributions

1. First Framework for Long-Horizon Transparent Manipulation: DeLTa is the first system to achieve precise, multi-step manipulation of novel transparent objects guided by language and video demonstration.
2. Single-Demonstration Generalization: The method eliminates the need for per-object demonstrations. By combining 6D pose estimation with canonicalized mesh retargeting, a single human video per skill (Pick/Place/Pour) transfers to novel objects.
3. Robust Perception Pipeline: Integrates foundation models for stereo depth estimation to solve the "transparent object depth" problem, enabling accurate 6D pose estimation where standard sensors fail.
4. Constraint-Aware VLM Planning: Introduces a novel planner that refines VLM outputs through iterative search and validation, specifically addressing single-arm and eye-in-hand constraints to prevent execution failures.
5. Real-World Validation: Demonstrates success in complex, cluttered environments (tight shelves, chemical labs) where existing methods fail.

---

4. Experimental Results

The system was evaluated on a Kinova Gen3 arm with an eye-in-hand ZED camera across three tasks:

• Task 1 (Tight Shelf Retrieval): Placing objects on a shelf.
• Task 2 (Chemical Experiment): Pouring liquids to specific colors (requires long-horizon planning and precision).
• Task 3 (Grocery Stocking): Arranging objects in a straight row (requires precise alignment).

Performance:

• Success Rates: DeLTa achieved 100% success on Task 1, 80% on Task 2, and 70% on Task 3.
• Baselines:
  • ClearGrasp (Depth-based): Failed significantly on pouring and alignment tasks due to noisy depth estimates on transparent surfaces.
  • YODO (Category-level Pose): Failed on novel/irregular objects and long-horizon tasks due to poor generalization outside trained categories and lack of collision checking.
• Ablation Studies:
  • Removing the enhanced depth estimation dropped success rates significantly (e.g., Task 3 dropped from 70% to 20%).
  • Using 3D position only (without 6D pose) caused failures in tasks requiring specific orientations (e.g., grocery stocking).
  • Removing the "Grounded Plan Search" (using naive VLMs) resulted in invalid plans that the robot could not execute.

---

5. Significance

DeLTa represents a major step forward in embodied AI for real-world applications.

• Bridging the Perception-Action Gap: It solves the specific optical challenges of transparent objects, a known bottleneck in robotics.
• Efficiency: By requiring only one demonstration per skill type rather than per object, it drastically reduces the data collection burden for deploying robots in dynamic environments (e.g., warehouses, labs, homes).
• Scalability: The integration of VLMs with constraint-aware search allows robots to understand complex natural language commands and execute them safely in cluttered, unstructured environments, moving beyond simple pick-and-place to complex assembly and organization tasks.

Limitations: The current system assumes rigid objects and relies on pre-built canonical meshes. Future work aims to extend this to deformable objects and automate the mesh generation pipeline.