Technical Summary: Pegasus – Physically Grounded Experience Synthesis for Embodied Intelligence

1. Problem Statement

The primary bottleneck in advancing embodied AI is not model architecture but the scarcity of high-quality, robot-specific training data. While billions of human manipulation videos exist online, robots cannot directly learn from them due to the embodiment gap: significant differences in morphology, kinematics, and action spaces between humans and robots prevent direct transfer of demonstrations.

Existing data acquisition paradigms face distinct limitations:

• Real-robot collection: High quality but expensive and difficult to scale.
• Physics simulation: Unlimited data but suffers from the sim-to-real gap, particularly for vision-based policies.
• Learning from human videos: Leverages abundant data but typically learns visual representations rather than generating executable robot trajectories.

The core challenge is to bridge this gap by transferring structured experience (task semantics, affordances, and constraints) rather than raw pixels, enabling the synthesis of robot-learnable data from human demonstrations.

2. Methodology: The Pegasus Framework

Pegasus is a low-resource framework that translates human egocentric videos into robot-executable demonstrations through a structured, graph-based pipeline. It reframes the problem from appearance generation to cross-embodiment knowledge transfer.

The framework operates in five progressive stages:

2.1 Task Graph Extraction ($V_h \rightarrow G_T$)

Instead of reasoning over raw pixels, Pegasus extracts a Task Graph ($G_T$) from human videos using a structured vision-language parser.

• Nodes: Represent semantic entities including sub-goals ($g_i$), manipulated objects ($o_j$), and spatial regions ($r_k$).
• Edges: Encode temporal ordering, causal dependencies, and spatial interactions.
• This graph preserves task topology while remaining independent of specific robot morphologies.

2.2 Hierarchical Affordance Latent ($G_T \rightarrow Z$)

To handle generalization across objects and embodiments, Pegasus introduces a Hierarchical Affordance Latent ($Z = \{Z_s, Z_a, Z_t\}$):

1. Object-State Level ($Z_s$): Captures physical states (e.g., grasped, opened, displaced).
2. Affordance Level ($Z_a$): Abstracts object-specific observations into manipulation capabilities (e.g., graspability, containment, pourability).
3. Task Level ($Z_t$): Represents embodiment-independent intentions (e.g., transporting, stacking).
   This hierarchy progressively removes embodiment-specific information while preserving causal task semantics. A contrastive objective aligns human and robot demonstrations within this shared affordance space.

2.3 Cross-Embodiment Graph Translation ($G_T \rightarrow G_P$)

The task graph is transformed into an executable Robot Planning Graph ($G_P$) through three graph operators:

1. Affordance Graph: Augments semantic nodes with manipulation affordances.
2. Constraint Graph: Incorporates embodiment-dependent feasibility (grasp types, inverse kinematics, collision constraints, workspace limits).
3. Planning Graph: Converts feasible primitives into executable trajectories (motion waypoints and temporal transitions).
   This modular formulation disentangles semantic reasoning from embodiment adaptation.

2.4 Physics-Constrained Verification

Visual plausibility is insufficient for robotics. Pegasus employs a closed-loop verification module that validates generated trajectories against:

• Inverse kinematics solvability
• Joint limits
• Collision avoidance
• Grasp stability and trajectory smoothness

If constraints are violated, structured error signals are fed back into the planning module for iterative regeneration until all constraints are satisfied.

2.5 Experience Rendering

The verified planning graph is rendered into three output modalities suitable for downstream policy learning:

• Video Demonstrations ($D_{video}$): For vision-language policies.
• Discrete Action Tokens ($D_{action}$): For autoregressive action models.
• Continuous Trajectories ($D_{trajectory}$): For classical control pipelines.

3. Key Contributions

1. Transfer Formulation: Recasting internet human videos as a scalable source of robot training data via structured cross-embodiment knowledge transfer.
2. Graph-Structured Framework: A novel pipeline that explicitly models task semantics, affordances, embodiment constraints, and robot planning, moving beyond raw pixel or text prompt conditioning.
3. Hierarchical Affordance Latent: A representation that improves generalization to unseen objects by organizing data according to object states, affordances, and tasks rather than object identity.
4. Closed-Loop Physics Verification: An iterative mechanism that enforces kinematic feasibility, converting embodiment adaptation into a constraint satisfaction problem.
5. Empirical Validation: Comprehensive experiments demonstrating improvements in task correctness, physical executability, and downstream policy learning.

4. Experimental Results

Pegasus was evaluated on benchmarks including GTEA Gaze+, EPIC-KITCHENS-100, and a curated Internet Manipulation Video Set, across five diverse robot embodiments (e.g., Franka Panda, xArm 7, UR5e).

4.1 Task Correctness and Executability

• Graph-conditioned generation outperformed prompt-only baselines by 12.3% in Task Correctness and 15.7% in Executability.
• Across five robot types, the system achieved an average Task Correctness of 84.4% (VLM) and 81.5% (Human), with 89.6% Executability.

4.2 Downstream Policy Learning

Training policies (OpenVLA and ACT) on generated data yielded significant results:

• Generated data alone achieved a 55.2% success rate, comparable to 84% of the performance achieved by 50 real-robot demonstrations (65.9%).
• Augmenting 50 real demonstrations with 200 generated ones improved the success rate by 11.3% absolute over real data alone.
• Compared to training on human video features (R3M/VIP), graph-conditioned generation provided a +25.8% improvement in success rate.

4.3 Generalization to Unseen Objects

The Affordance Latent demonstrated strong zero-shot capabilities:

• Achieved 78.5% task correctness on completely unseen objects (e.g., watermelon, screwdriver) compared to 45.1% for object-identity-based latents.
• The model successfully inferred necessary behaviors (e.g., "heavy + fragile" $\rightarrow$ "careful power grasp") based on affordance composition.

4.4 Physics Validity

The closed-loop verification significantly improved physical feasibility:

• Pass@1 (first attempt) was 51.2%; Pass@5 (after 5 iterations) reached 94.1%.
• Inverse Kinematics (IK) error was reduced by 93.5%, and collision rates dropped by 90.9%.

4.5 Ablation Studies

• Removing the Constraint Graph caused the largest drop in Executability (-11.2%).
• Removing the Affordance Graph caused the largest drop in Task Correctness on unseen objects (-5.1%).
• The graph structure was identified as the primary driver of performance, rather than the underlying video generation model.

5. Significance and Claims

The paper claims that Pegasus successfully reframes robot data generation from a hardware collection problem into a scalable, low-resource knowledge transfer problem.

• Experience-Level Transfer: The work argues that the embodiment gap should be bridged at the experience level (goals, state transitions, affordances) rather than the pixel level. This allows the system to leverage the diversity and realism of human videos without requiring large-scale 3D asset construction or simulation.
• Scalability: By decoupling task semantics from robot morphology, the framework enables the synthesis of robot-learnable demonstrations for heterogeneous platforms from a single source of human data.
• Practicality: The results suggest that explicitly modeling task knowledge and affordance structures allows the vast collection of human manipulation videos to serve as effective supervision for robot learning, offering a practical alternative to collecting every demonstration on physical hardware.

The authors acknowledge limitations, including dependence on VideoLLMs for task decomposition, the focus on kinematic consistency rather than full contact dynamics, and current constraints to single-view generation. However, the framework establishes a new paradigm for structured cross-embodiment representation learning.