FeasibleCap: Real-Time Embodiment Constraint Guidance for In-the-Wild Robot Demonstration Collection

FeasibleCap is a gripper-in-hand data collection system that provides real-time, hardware-free executability feedback through visual and haptic cues, enabling demonstrators to capture valid robot trajectories without headsets or learned dynamics models while improving replay success and preserving cross-embodiment transfer.

The Gist

Imagine you are teaching a robot how to do a task, like picking up a block or tossing a ball into a bin. Usually, you would have to use the actual robot to teach it, which is slow, expensive, and risky if the robot breaks.

To solve this, researchers developed a method called "Gripper-in-Hand." Instead of using the robot, you hold a robotic hand (a gripper) in your own hand and move it around like you're doing the task. The computer records your movements, and later, they try to play those movements back on the real robot.

The Problem: The "Surprise" Failure
Here's the catch: When you hold the gripper, you don't know the robot's physical limits.

• Maybe you move your hand too fast, and the robot's joints can't spin that quickly.
• Maybe you move your hand to a spot the robot's arm physically can't reach.
• Maybe you move in a way that would make the robot crash into itself.

In the old system, you wouldn't find out about these problems until after you finished recording. You'd try to play the video back on the robot, and boom—it fails. Now you have to throw away that recording, start over, and try again. It's like trying to bake a cake without an oven, only to realize halfway through that you forgot the flour. You have to start the whole process over.

The Solution: FeasibleCap (The "Smart Coach")
The paper introduces FeasibleCap, a system that acts like a real-time coach for your hand.

Here is how it works, using a simple analogy:

1. The Setup: The "Magic Screen"

They take a standard iPhone and mount it onto the robotic gripper you are holding.

• The Camera looks out at the world.
• The Screen faces you, the human.

2. The Ghost Arm (The "Mirror")

As you move the gripper, the iPhone uses its advanced sensors (ARKit) to track your hand's position instantly. It then draws a ghostly, transparent version of the robot's arm right on your screen, overlaid on top of the real world.

Think of this like a dance instructor standing next to you. As you move, the instructor mimics your moves but is also checking if you are doing them correctly.

3. The Traffic Light System (The Feedback)

This is the magic part. The system constantly checks three things in real-time:

1. Reachability: "Can the robot actually reach that spot?"
2. Speed Limits: "Are you moving too fast for the robot's joints?"
3. Collisions: "Are you about to hit the robot's own body?"

It gives you immediate feedback using a Traffic Light System:

• 🟢 Green (Feasible): The ghost arm is green. You are moving perfectly within the robot's limits. Keep going!
• 🟡 Yellow (Warning): The ghost turns yellow and the phone vibrates gently. "Hey, you're getting close to the edge. Slow down or adjust slightly."
• 🔴 Red (Infeasible): The ghost turns red and vibrates strongly. "Stop! You are moving in a way the robot physically cannot do."

4. The Result: No More Surprises

Because you get this feedback while you are moving, you can correct your hand instantly. If you see the light turn red, you slow down or change your path immediately.

Why is this a big deal?

• Before: You record 10 videos, try to play them back, and maybe only 2 work. You wasted time on the other 8.
• With FeasibleCap: You record 10 videos, and because you were guided by the "traffic light," all 10 work when played back on the robot.

The "Tossing" Test

The researchers tested this with two tasks:

1. Picking up a block: This is slow and easy. The system helped a little, but it wasn't a huge game-changer.
2. Tossing a ball: This requires fast, jerky movements. Without the system, people moved too fast, and the robot failed 8 out of 10 times. With FeasibleCap, the success rate jumped to 6 out of 10 (and the failures were mostly just tiny, unavoidable physical limits).

The Bottom Line

FeasibleCap turns robot training from a "trial and error" guessing game into a guided learning experience. It allows humans to collect high-quality data without needing the expensive robot present during the recording, and without needing fancy VR headsets. It's like giving a student a tutor who whispers corrections in their ear while they take the test, rather than waiting until the test is graded to tell them what they got wrong.

Technical Summary

Here is a detailed technical summary of the paper "FeasibleCap: Real-Time Embodiment Constraint Guidance for In-the-Wild Robot Demonstration Collection."

1. Problem Statement

The "Executability Gap" in Robot-Free Data Collection:
Gripper-in-hand data collection allows researchers to gather large-scale demonstration datasets without requiring physical robot hardware during the capture phase. While this decouples data collection from robot resources, it introduces a critical flaw: the demonstrator has no awareness of the target robot's kinematic and dynamic constraints (e.g., workspace boundaries, joint-rate limits, self-collisions).

• Consequence: Demonstrations are often collected as "open-loop" trajectories. Feasibility is only discovered during a separate, costly "replay-and-validate" stage.
• Impact: Failed demonstrations incur the full cost of collection, diagnosis, and re-collection. This is particularly severe for dynamic tasks (like tossing) where joint-rate limits are easily exceeded, leading to a high rate of unusable data and inflated effective costs.
• Limitations of Existing Solutions: Prior work (e.g., ARCap, ARMADA) provides real-time feedback but relies on head-worn AR/VR displays, robot-in-the-loop hardware, or learned dynamics models. These are incompatible with the lightweight, retargeting-free, robot-free gripper-in-hand paradigm.

2. Methodology: FeasibleCap System

FeasibleCap is a system designed to bring real-time executability guidance into robot-free capture without headsets, external robots, or learned models.

A. Hardware Architecture

The system consists of three layers:

1. Handheld Device: An iPhone (e.g., iPhone 15 Pro Max) mounted on a modular handheld gripper platform (RAPID). The camera faces outward for tracking, and the screen faces the demonstrator for feedback.
2. Edge Compute Node: A Raspberry Pi 5 handles sensor synchronization, data logging (MCAP format), and replay command dispatch via a manufacturer's API.
3. Communication: The iPhone streams pose and image data to the Pi over WiFi (TCP) for asynchronous logging, while the feasibility loop runs entirely on-device to ensure low latency.

B. Core Algorithm: On-Device Feasibility Pipeline

Running at 60 Hz on the iPhone, the pipeline performs the following steps for every frame:

1. Pose Estimation: Uses ARKit VIO to estimate the 6-DoF end-effector pose ($p_t$).
2. Inverse Kinematics (IK): Solves for joint angles ($q_t$) using a Damped Least Squares (DLS) solver, warm-started from the previous frame's solution for stability.
3. Constraint Checking:
   • Reachability: Verifies if an IK solution exists.
   • Joint-Rate Limits: Calculates joint velocities ($\dot{q}$) from consecutive frames and checks against maximum limits.
   • Self-Collision: Checks for collisions between non-adjacent links using simplified geometric shapes (capsules/spheres).
4. Feedback Generation:
   • Visual: Renders a "ghost arm" (virtual robot) overlaid on the live camera feed via SceneKit.
     • Green: Feasible.
     • Yellow: Warning (approaching limits).
     • Red: Infeasible (violation detected).
   • Haptic: Delivers vibration cues via CoreHaptics (intermittent for warnings, continuous for infeasibility).

C. Workflow

• Calibration: Users perform a one-shot visual alignment to calibrate the transform between the camera and the gripper's Tool Center Point (TCP).
• Clutch Mechanism: A software clutch allows the user to freeze the virtual ghost arm to inspect alignment without generating motion data.
• Closed-Loop Collection: The demonstrator adjusts their motion in real-time based on the visual/haptic cues to stay within the executable region.
• Replay: The recorded trajectory is played back on the target robot (Realman RM75) via the Raspberry Pi.

3. Key Contributions

1. Identification of the Gap: The authors identify that robot-free collection pipelines lack real-time feedback mechanisms, leading to high costs from replay failures.
2. FeasibleCap System: The first system to integrate real-time executability feedback into the gripper-in-hand paradigm without head-mounted displays, robot hardware, or learned dynamics models. It relies purely on analytical kinematic constraints derived from the robot's URDF.
3. Empirical Validation: Demonstrated that real-time guidance significantly improves replay success rates, particularly for dynamic tasks, while preserving cross-embodiment transferability.

4. Experimental Results

Experiments were conducted on Pick-and-Place and Tossing tasks using a Realman RM75 robot.

• Replay Success Rate:

  • Pick-and-Place: Improved from 80% (Baseline) to 100% (FeasibleCap).
  • Tossing: Improved from 20% (Baseline) to 60% (FeasibleCap).
  • Overall: Increased from 50% to 80% success rate.
  • Insight: The gains were most significant for tossing, where joint-rate violations are common and invisible to unguided users.

• Feasibility Analysis (Frame-Level):

  • Baseline: Trajectories contained high ratios of infeasible frames (e.g., 56% for pick-and-place, 53% for tossing).
  • FeasibleCap: Reduced infeasible frames to 0% for pick-and-place and 14% for tossing.
  • Failure Modes: Baseline failures were distributed throughout the trajectory due to exaggerated motions. FeasibleCap failures were concentrated only at the "release" instant of the toss, where physical dynamics inherently exceed robot limits (a transient spike too fast for human reaction).

• Cross-Embodiment Transferability:

  • Demonstrations collected with a Franka Panda constraint model successfully replayed on a Realman RM75 (7/10 success).
  • Demonstrations collected with RM75 constraints successfully replayed on Franka Panda in simulation (8/10 success).
  • Conclusion: Enforcing constraints for one robot does not over-specialize the data, preserving transferability to other 7-DoF platforms.

5. Significance and Impact

• Cost Reduction: By preventing the collection of infeasible data, FeasibleCap drastically reduces the "cost per usable trajectory," making large-scale data collection more efficient.
• Accessibility: It democratizes high-quality data collection by removing the need for expensive AR headsets or active robot hardware during the capture phase.
• Data Quality: It shifts the paradigm from "collect then filter" to "collect correctly," ensuring that the resulting datasets are inherently executable and safer for training policies.
• Future Directions: The authors suggest extending feedback to continuous scores (rather than discrete states) and supporting bimanual manipulation.

In summary, FeasibleCap bridges the gap between human demonstration and robot execution by providing immediate, hardware-agnostic kinematic feedback, enabling the collection of high-fidelity, executable robot data in the wild.