💻 computer science

PAWS: Perception of Articulation in the Wild at Scale from Egocentric Videos

PAWS is a novel method that addresses the scalability limitations of existing articulation perception techniques by directly extracting object motion and structure from large-scale, in-the-wild egocentric videos through hand-object interactions, demonstrating significant improvements on public datasets and benefits for downstream robotics and 3D prediction tasks.

Original authors: Yihao Wang, Yang Miao, Wenshuai Zhao, Wenyan Yang, Zihan Wang, Joni Pajarinen, Luc Van Gool, Danda Pani Paudel, Juho Kannala, Xi Wang, Arno Solin

Published 2026-03-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yihao Wang, Yang Miao, Wenshuai Zhao, Wenyan Yang, Zihan Wang, Joni Pajarinen, Luc Van Gool, Danda Pani Paudel, Juho Kannala, Xi Wang, Arno Solin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are wearing a smart camera on your head, recording your day as you go about your life. You open the fridge, pull out a drawer, and close a cabinet door. To a human, these actions are obvious: "That's a door that swings," or "That's a drawer that slides."

But to a computer, a video is just a chaotic stream of pixels. It doesn't know that the thing you're touching is a "cupboard" or that it moves on a "hinge." It just sees a blurry shape moving.

PAWS (Perception of Articulation in the Wild at Scale) is a new AI system designed to solve this problem. It teaches computers to watch your "wild" (unscripted, messy, real-life) videos and figure out exactly how the furniture in your house moves, without needing a lab or a team of humans to label every single frame.

Here is how it works, using some everyday analogies:

1. The Problem: The "Blind" Robot

Currently, if you want a robot to open your fridge, you usually have to give it a perfect 3D map of the fridge and tell it exactly where the handle is. This is like giving a robot a blueprint of a house before it's even built. It's expensive, slow, and doesn't work well in the real world where things are messy, dark, or partially hidden.

Existing AI methods try to guess how things move by looking at static pictures, but they often get it wrong because they lack context. They are like someone trying to guess how a door opens by looking at a single photo of a closed door—they might guess it slides, when it actually swings.

2. The Solution: PAWS (The "Sherlock Holmes" of Motion)

PAWS takes a different approach. Instead of looking at static photos, it watches you interacting with the object. It uses two main clues:

The Hand as a Detective: When you open a drawer, your hand moves in a straight line. When you open a door, your hand moves in a circle. PAWS tracks your hand like a detective following a trail of breadcrumbs. It ignores the messy background and focuses entirely on how your fingers move relative to the object.
The "Brain" (VLM): PAWS uses a "Vision-Language Model" (a super-smart AI that understands both images and text). Think of this as a knowledgeable librarian. If the video shows a hand pulling a handle and the text says "open the cupboard," the librarian says, "Ah! Cupboards usually swing on hinges, they don't slide!" This helps the AI make the right guess even if the video is blurry.

3. How It Works: The Three-Step Dance

Step 1: The "Hand Dance" (Dynamic Interaction)
PAWS looks at the video and finds the parts where you are touching something. It reconstructs your hand in 3D space.

Analogy: Imagine your hand is a dancer. If the dancer moves in a straight line, PAWS knows the object is a sliding drawer. If the dancer moves in a curve, PAWS knows it's a swinging door.

Step 2: The "Room Scan" (Static Geometry)
While your hand is dancing, PAWS also looks at the room around you to build a 3D map of the furniture. It looks for straight lines and corners (like the edges of a cabinet).

Analogy: This is like a carpenter looking at the wood grain and the angles of a table to understand how it was built. It helps PAWS know, "Okay, this object is aligned with the wall, so it probably slides along that wall."

Step 3: The "Consultation" (VLM Reasoning)
Sometimes, the hand movement is confusing (maybe you slipped, or the video is dark). This is where the "Librarian" (the VLM) steps in.

Analogy: PAWS asks the AI: "I see a hand near a white box. The hand moved in a circle. Is this a microwave or a cabinet?" The AI replies: "Microwaves usually have doors that swing, but they are small. This looks like a big cabinet. Let's assume it's a swinging door."

4. Why This Matters: From "Watching" to "Doing"

The coolest part is what happens after PAWS figures it out.

Teaching Robots: Once PAWS understands how a specific cupboard in a specific kitchen works, it can teach a real robot (like the Boston Dynamics Spot robot mentioned in the paper) how to open it. The robot can then go into a new house and open the doors without needing a manual.
Better Simulations: It helps video game developers and movie animators create realistic 3D worlds where objects move exactly like they do in real life.

The Big Picture

Before PAWS, teaching a computer how the world moves required expensive 3D scanners and perfect lighting. PAWS is like giving the computer a pair of eyes and a brain that can learn from billions of hours of real people's home videos.

It turns the chaos of "in-the-wild" videos (where people are messy, lighting is bad, and cameras shake) into a clean, mathematical understanding of how our world is put together. It's the difference between a robot that needs a manual for every door it encounters, and a robot that just watches you open a door once and then knows how to do it forever.

1. Problem Statement

Articulation perception involves recovering the motion and structure of articulated objects (e.g., drawers, cupboards, fridge doors) to enable 3D scene understanding for robotics, simulation, and animation.

Current Limitations: Existing learning-based methods rely heavily on supervised training with high-quality, manually annotated 3D data. This limits scalability and diversity.
The Gap: While "in-the-wild" egocentric videos (first-person videos) offer massive scale and natural human-object interactions, they are monocular, noisy, unstructured, and lack the multi-view or depth data required by traditional articulation inference methods.
Goal: Develop a method to automatically extract scene-level articulation parameters (motion type, axis, and origin) directly from untrimmed, monocular egocentric RGB videos without task-specific training or high-fidelity 3D ground truth.

2. Methodology: PAWS Pipeline

PAWS is a training-free, fully automatic pipeline that leverages hand-object interactions and foundation model priors. It consists of four core stages:

A. Dynamic Interaction Perception

Input: An egocentric video clip and a language description of an action (e.g., "open the cupboard").
Hand Reconstruction: Uses a frame-based 3D hand reconstruction method (MANO model) to extract 3D hand trajectories.
Trajectory Refinement: Raw hand trajectories are noisy. PAWS applies Ornstein-Uhlenbeck (OU) processes and Rauch–Tung–Striebel (RTS) smoothing to filter out pre/post-interaction noise, ensuring the trajectory represents physical contact with the object.
Coarse Localization: Reconstructs a coarse 3D voxel grid of the scene from the interaction clip to identify high-confidence regions where the object is located.

B. Static Scene Geometry Recovery

View Reselection: Selects a subset of "global views" from the full video that maximize spatial coverage of the object while avoiding occlusions.
Revolute Motion (Hinges): Uses multi-view line triangulation. It detects 2D lines, unprojects them to 3D using metric depth estimation, and fits 3D line models (candidates for rotation axes) using LO-RANSAC.
Prismatic Motion (Sliding): Leverages the Manhattan World Assumption. It estimates dominant orthogonal directions (room/furniture axes) from surface normals and depth maps to serve as candidates for sliding directions.

C. Vision-Language Guided Motion Reasoning

To resolve ambiguity between geometric candidates and noise, PAWS employs a Vision-Language Model (VLM) in a two-stage process:

Temporal Motion Type Classification: The VLM analyzes sparse frames from the interaction clip to classify the motion type as either Revolute (rotation) or Prismatic (translation) based on visual cues and language context.
Spatial Axis Grounding: The VLM is presented with the 3D line candidates projected onto a 2D image (as numbered "Set-of-Marks"). It uses physical commonsense to select the most plausible axis (e.g., identifying a door hinge vs. a table edge).

D. Joint Articulation Inference

Integration: Combines the refined hand trajectories (dynamic cues) with the geometric candidates (static cues).
Optimization:
- For Prismatic motion: Selects the Manhattan axis best aligned with the hand's primary motion direction.
- For Revolute motion: Selects the 3D axis that minimizes the variance of the orthogonal distance to the hand trajectory (enforcing a consistent rotation radius).
Output: Final articulation parameters $\phi = \{c_i, a_i, o_i\}$ (Motion Type, Axis, Origin).

3. Key Contributions

Training-Free Pipeline: Introduces the first fully automatic method for scene-level articulation perception from monocular, in-the-wild egocentric videos without requiring task-specific supervised training.
Foundation Model Integration: Proposes a novel approach combining VLMs (for semantic reasoning and axis selection) and LLMs (for aggregating temporal context) to infer plausible kinematics where geometric data is ambiguous.
Scalable Data Generation: Demonstrates the ability to generate high-quality articulation annotations at scale from existing large-scale datasets (HD-EPIC, Epic-Fields), overcoming the bottleneck of manual 3D annotation.
Downstream Utility: Validates that the extracted articulations significantly improve downstream tasks, including fine-tuning 3D articulation prediction models and enabling real-world robotic manipulation.

4. Experimental Results

The method was evaluated on HD-EPIC, Arti4D, and Epic-Fields datasets.

Performance vs. Baselines: PAWS significantly outperforms strong baselines (Articulation3D, ArtiPoint, Articulate-Anything, iTACO) on in-the-wild data.
- On HD-EPIC, PAWS achieved a 71.43% match rate (scene-level aggregation), compared to 42.22% for the best baseline (Articulation3D++).
- PAWS is particularly robust to the domain gap of egocentric videos (rapid viewpoint changes, occlusions) where static-input or multi-view methods fail.
Ablation Studies:
- Removing hand contact constraints reduced accuracy, proving the necessity of dynamic interaction cues.
- Removing VLM reasoning dropped performance, confirming that geometric priors alone are insufficient for disambiguating complex scenes.
Downstream Applications:
- Fine-tuning: Using PAWS-generated labels to fine-tune USDNet improved articulation prediction accuracy on both in-domain and cross-dataset benchmarks.
- Robotics: The extracted parameters were successfully used to control a Boston Dynamics Spot robot to open/close cupboards and drawers in a real-world setting using Dynamic Movement Primitives (DMP).

5. Significance

PAWS represents a paradigm shift in articulation perception by moving away from expensive, controlled 3D data collection toward leveraging the abundance of unstructured human video.

Scalability: It unlocks the potential of "in-the-wild" data for training and testing 3D scene understanding models.
Robotic Deployment: By providing a way to infer 3D kinematic structures from simple RGB videos, it bridges the gap between human demonstration and robotic execution, allowing robots to learn manipulation skills from everyday videos without explicit 3D sensing or manual annotation.
Generalization: The method's reliance on foundation model priors and geometric constraints rather than specific object categories allows it to generalize to novel objects and scenes.