💻 computer science

CORE: Common Outcome Regularities from Action-Free Visual Demonstrations for Robot Manipulation

The paper introduces CORE, a robot imitation learning framework that leverages action-free human videos by extracting and utilizing common terminal outcome regularities as visual goal prototypes to overcome embodiment gaps and significantly improve manipulation success rates across simulated and real-world tasks.

Original authors: Juyi Sheng, Jincheng Li, Mingxin Tan, Mengyuan Liu

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Juyi Sheng, Jincheng Li, Mingxin Tan, Mengyuan Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to stack bowls. You have two ways to give it instructions:

The Text Manual: You tell the robot, "Please stack the bowls."
The Video: You show the robot a video of a human stacking bowls, but you don't tell the robot how to move its arms to do it. You just let it watch the result.

For a long time, robots have struggled with the second option. They are great at following text instructions, but text is often too vague. "Stack the bowls" doesn't tell the robot exactly how high the stack should be, how the bowls should touch, or what the final shape needs to look like. It's like telling a chef to "make a cake" without showing them what a finished cake looks like.

On the other hand, robots have also struggled to learn from human videos because humans and robots look very different. A human uses hands; a robot uses a gripper. A human moves their whole body; a robot moves a specific arm. Trying to copy a human's exact hand movements usually confuses the robot.

The Big Idea: "The Finish Line, Not the Race"

The authors of this paper, CORE, realized something clever. They noticed that while different people might stack bowls in totally different ways (using different speeds, different angles, or different hand grips), they all end up with the exact same result: a neat, stable stack of bowls.

They call these shared results "Common Outcome Regularities."

Instead of trying to teach the robot how to move (the race), they decided to teach the robot what the finish line looks like (the result).

How CORE Works (The Three Steps)

Think of CORE as a smart teacher who watches a bunch of videos and then gives the robot a "Goal Picture."

Step 1: The Detective (Learning the Result)
The system watches many videos of people successfully stacking bowls. It ignores the messy middle parts of the video (the waving arms, the pauses) and focuses only on the very last second of the successful attempts. It learns to recognize the "fingerprint" of a successful stack. It's like a detective who only cares about the solved crime scene, not the journey to get there.
Step 2: The Artist (Creating the Goal)
The system takes all those "successful ending" snapshots and blends them together to create one perfect, ideal "Goal Picture." This isn't just one random photo; it's a summary of what a perfect stack looks like, filtering out any weird or accidental ways people might have finished the task.
Step 3: The Coach (Guiding the Robot)
Now, the robot tries to do the task. As it moves, the system constantly compares what the robot is seeing right now with that "Goal Picture." It tells the robot, "You are getting closer to the Goal Picture," or "You are moving away from it." This acts like a GPS that constantly corrects the robot's path until it matches the perfect ending.

Why This is Better Than Text

The paper tested this on many different tasks, like opening drawers, stacking blocks, and moving objects.

Text Instructions are like a vague map: "Go to the park." The robot might get lost because it doesn't know exactly where the park bench is or how to climb the fence.
CORE's Visual Goals are like a photo of the destination: "Stop exactly when you look like this."

In their tests, robots using CORE were much more successful than those using text instructions.

In a simulation called Meta-World, robots improved their success rate by about 4%.
In RoboTwin 2.0, they improved by 11%.
In the Real World (using a physical robot arm), the improvement was huge: 17%.

The Real-World Proof

The researchers even tried this on a real robot arm in a lab. They asked the robot to open a drawer and stack three bowls.

Robots using only text instructions often stopped just short of the target or fumbled the stack.
Robots using CORE's "Goal Picture" knew exactly when they had reached the perfect position and completed the task successfully.

The Takeaway

The paper argues that we don't need to teach robots to copy human movements exactly. Instead, we should teach them to recognize what a "good job" looks like. By focusing on the outcome rather than the action, robots can learn from the vast amount of human videos available on the internet, even if they don't look like humans. It's a shift from asking "How do I move?" to asking "What does success look like?"

Technical Summary: CORE – Common Outcome Regularities from Action-Free Visual Demonstrations

Problem Statement

Robot imitation learning currently faces a scalability bottleneck: high-performance policies typically require large amounts of costly, robot-specific demonstrations (e.g., teleoperation or kinesthetic teaching). While action-free visual demonstrations, such as human videos from the internet, are abundant and rich in task priors, they are difficult to utilize directly. These videos lack robot-executable action labels and suffer from significant "embodiment gaps" in morphology, action spaces, and control interfaces.

Furthermore, existing methods that attempt to condition policies on task goals often rely on natural language instructions. While language conveys high-level intent (e.g., "stack the bowls"), it frequently fails to specify the precise geometric, spatial, and contact constraints required for successful execution in contact-rich manipulation tasks. This under-specification can lead to sub-optimal policies that fail to achieve the necessary terminal states.

Methodology: The CORE Framework

The authors propose CORE (Common Outcome Regularities), a policy learning framework that extracts shared structural regularities from successful terminal states in action-free visual demonstrations, rather than attempting to transfer explicit actions or rely solely on language. CORE operates in three distinct stages:

Stage I: Terminal Outcome Representation Learning
The framework trains a terminal outcome encoder ( $f_\phi^z$ ) using action-free visual demonstrations. The encoder is optimized with a terminal contrastive objective to pull together terminal observations from successful demonstrations of the same task while pushing apart non-terminal or unrelated outcomes. This is augmented by two auxiliary tasks:
- Time-to-Goal Regression ( $L_{ttg}$ ): Encourages the representation to be aware of temporal progress toward the goal.
- Terminal Classification ( $L_{term}$ ): Improves discrimination between terminal and non-terminal states.
  The goal is to learn representations that focus on shared terminal structures (object configurations, spatial relations) rather than trajectory-specific motion details.
Stage II: Goal Prototype Construction
Successful terminal embeddings are aggregated to form a visual goal prototype ( $z_{goal}$ ). To ensure robustness, the method clusters successful terminal embeddings using K-Means and selects the dominant modes (Top-K) to compute a static, task-level goal condition. This aggregation suppresses noisy or rare terminal observations, yielding a stable representation of the desired physical end state.
Stage III: Backbone-Agnostic Goal Injection
During policy training and inference, the current observation is encoded into an outcome embedding ( $z_t$ ). This is concatenated with the pre-computed goal prototype ( $z_{goal}$ ) to form a global goal feature ( $g_t = [z_t; z_{goal}]$ ). This feature is injected into the robot policy backbone (e.g., Diffusion Policy, MP1) via a conditioning operator. The policy is then guided to generate action chunks that minimize the distance between the current state and the desired terminal state. This module is designed to be compatible with various policy backbones, including diffusion-based and flow-based models.

Key Contributions

Formulation of Outcome-Level Learning: The paper reframes visual-demonstration learning as the extraction of Common Outcome Regularities. Instead of struggling to map explicit actions across different embodiments, CORE extracts shared structures directly from successful terminal states.
Three-Stage Framework: The authors propose a novel pipeline that converts action-free visual data into robust visual goal conditions through terminal representation learning and prototype construction. The design is backbone-agnostic, allowing seamless integration into diverse downstream policies.
Superiority over Language Conditioning: The work demonstrates that visual goal prototypes provide more concrete geometric and physical constraints than text instructions, particularly for tasks requiring precise terminal-state alignment.

Experimental Results

The authors evaluated CORE on two simulation benchmarks (Meta-World and RoboTwin 2.0) and a real-world robotic arm setup (ARX R5).

Meta-World: CORE improved the average success rate of the MP1 backbone by +3.9% (from 81.7% to 85.6%) and the DP3 backbone by +2.5% (from 81.5% to 84.0%). Improvements were more pronounced on "Hard" and "Very Hard" tasks. Text-conditioned variants showed negligible gains in comparison.
RoboTwin 2.0: CORE yielded significant improvements, boosting the MP1 backbone by +11.1% (to 79.4%) and the DP3 backbone by +4.8% (to 82.3%). In every evaluated task, at least one CORE-integrated policy achieved the best result.
Real-World: On a physical robot performing contact-rich tasks (e.g., drawer manipulation, stacking), CORE improved the MP1 backbone's success rate by +17.0% (from 69.0% to 85.0%) and DP3 by +17.0% (from 65.0% to 82.0%). Text-conditioned baselines provided negligible improvements in the real-world setting.
Ablation Studies: Removing auxiliary losses ( $L_{ttg}, L_{term}$ ) or the current outcome embedding ( $z_t$ ) significantly degraded performance, confirming the necessity of the full three-stage design. The study also showed that learning a specialized terminal encoder outperforms using frozen, general-purpose visual encoders (e.g., Uni3D).

Significance and Claims

The paper claims that the central challenge in utilizing human videos is not merely data availability, but extracting supervision that is usable by robot policies despite embodiment gaps. CORE addresses this by shifting the goal representation from "under-specified language intent" to "outcome-level goal conditions induced from successful observations."

The authors assert that by focusing on Common Outcome Regularities, the framework allows robots to leverage abundant, action-free human data without needing to recover explicit actions or bridge morphological gaps directly. The results suggest that visual goal prototypes offer a more direct and physically grounded signal for task completion than language, enabling more robust manipulation policies even with limited robot-specific demonstration data.