💻 computer science

FunRec: Reconstructing Functional 3D Scenes from Egocentric Interaction Videos

FunRec is a novel method that reconstructs functional, simulation-ready 3D digital twins of indoor scenes directly from in-the-wild egocentric RGB-D interaction videos by automatically discovering articulated parts and their kinematics, significantly outperforming existing approaches in segmentation, pose accuracy, and reconstruction quality.

Original authors: Alexandros Delitzas, Chenyangguang Zhang, Alexey Gavryushin, Tommaso Di Mario, Boyang Sun, Rishabh Dabral, Leonidas Guibas, Christian Theobalt, Marc Pollefeys, Francis Engelmann, Daniel Barath

Published 2026-04-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Alexandros Delitzas, Chenyangguang Zhang, Alexey Gavryushin, Tommaso Di Mario, Boyang Sun, Rishabh Dabral, Leonidas Guibas, Christian Theobalt, Marc Pollefeys, Francis Engelmann, Daniel Barath

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you walk into a room and look at a closed cabinet. A standard 3D scanner sees a box. It knows the shape, the color, and the size. But it doesn't know that the door can swing open, that the drawer can slide out, or that the handle is the key to unlocking the whole thing. It's like taking a photo of a car and only seeing a metal shell, missing the engine, the wheels, and the fact that it can drive.

FunREC is a new technology that changes the game. Instead of just taking a "snapshot," it watches you interact with the world to build a "living" 3D model.

Here is how it works, broken down with some everyday analogies:

1. The Problem: The "Frozen World"

Most 3D maps we have today are like frozen statues. They capture a scene exactly as it is at one moment. If you open a fridge door in the real world, the 3D map doesn't know the door can open. It just sees a wall. This is a problem for robots or AI because they need to know how things move to do their jobs.

2. The Solution: The "Detective with a Camera"

FunREC is like a super-smart detective wearing a camera on their head (like a GoPro). It doesn't just watch; it watches you do things.

The Input: You wear the camera, walk around your apartment, open a drawer, swing a door, and pull out a chair.
The Magic: As you move, the system asks: "Wait, that part moved while that part stayed still. That means they are connected by a hinge or a slider!"

3. How It Builds the "Digital Twin"

Think of the system as a master LEGO builder who is trying to rebuild your room using only a video of you playing with it.

Step 1: The "Movie Clip" Breakdown
The video is chopped up into tiny scenes. The system uses AI to spot the "action" parts (like when you grab a handle) and ignores the boring parts (like when you just walk past a wall).
Step 2: The "Motion Detective"
It tracks every single point on the objects. It notices: "Hey, all these points on the drawer moved in a straight line. That's a sliding joint." Or, "These points on the door moved in a circle. That's a hinge."
It's like watching a dance and figuring out who is the lead and who is the follower just by how they move.
Step 3: The "Ghost Builder"
Once it knows how things move, it builds two versions of the room:
1. The Static Room: The walls, the floor, and the furniture that doesn't move.
2. The "Ghost" Parts: It builds a perfect 3D model of the moving parts (the door, the drawer) in their "home" position (closed).
  Then, it figures out the exact mathematical rules (the "physics") that tell the door how to swing or the drawer how to slide.

4. The Result: A "Video Game" Version of Your Room

The final output isn't just a picture; it's a simulation-ready 3D world.

Interactive: You can click the door in the computer, and it will swing open exactly like it did in real life.
Robots Can Use It: If you send a robot into this digital room, the robot knows, "Oh, I need to grab the handle and pull down to open this," because the digital twin told it exactly how the joint works.
No Manual Work: You don't have to tell the computer "This is a door." The computer figured it out just by watching you open it.

Why Is This a Big Deal?

Previously, to get a robot to open a door, engineers had to:

Manually measure the door.
Program the hinge location.
Test it a hundred times.

With FunREC, you just walk around, open a few things, and the computer does all the math. It turns a "frozen" photo of a room into a playable, interactive video game that a robot can learn from.

In short: FunREC turns a boring video of you doing chores into a magical, interactive 3D map that knows exactly how every door, drawer, and lid works.

1. Problem Definition

The paper addresses the challenge of reconstructing functional 3D digital twins of indoor environments directly from single-view, egocentric (first-person) RGB-D interaction videos.

The Gap: Existing 3D reconstruction methods typically produce static scenes. While some recent works attempt to model articulated objects (e.g., doors, drawers), they often rely on:
- Controlled setups with multi-state captures (scanning the same room open and closed).
- Manual annotations or CAD priors.
- Object-centric views rather than full scene-scale reconstruction.
- Static cameras, whereas real-world interaction is dynamic and egocentric.
The Goal: To automatically discover articulated parts, estimate their kinematic parameters (joint axes, types), track their 3D motion, and reconstruct both static geometry and moving parts into a simulation-ready, physically consistent 3D scene without manual intervention or pre-scanned models.

2. Methodology (FunREC)

FunREC is a training-free, optimization-based pipeline that integrates geometric reasoning with semantic and motion priors from foundation models. The process is divided into four main stages:

A. Fragment Construction & Classification

The input video is divided into temporally contiguous fragments.
A Video-Language Model (VLM) classifies each fragment as static (no interaction) or dynamic (interaction with an articulated part).
For dynamic fragments, the VLM predicts the articulation type: Prismatic (sliding, e.g., drawers) or Revolute (rotating, e.g., doors).

B. Dynamic Fragment Reconstruction

This stage processes each dynamic fragment independently:

Camera Pose Estimation: Dense point correspondences are extracted using RoMA. Matches are filtered using hand and object masks (from VISOR) to discard hand-occluded points. Relative camera motion is estimated via SupeRANSAC, followed by pose graph optimization for global consistency.
Sparse 3D Trajectories: Sparse point tracks are generated in the world frame using TAPIP3D. New tracks are initialized to capture newly revealed surfaces.
Articulation-Aware Motion Clustering:
- Tracks are fitted to 3D lines (prismatic) or circles (revolute) to hypothesize joint parameters.
- Tracks are filtered based on fitting error and clustered using HDBSCAN based on similarity in joint parameters (axis, pivot, motion pattern).
- The cluster corresponding to the manipulated part is selected by maximizing consistency with 2D interaction masks.
Pixel-Aligned Part Segmentation:
- Keyframes are selected, and SAM (Segment Anything Model) generates over-segmentation masks.
- The ratio of moving vs. static tracks within each semantic region determines if the region belongs to the moving part.
- These masks are propagated temporally using SAM2 to create dense, consistent segmentation masks for the articulated part.
Joint Pose & Articulation Optimization:
- A non-linear optimization (using Ceres Solver) jointly refines the part poses ( $T^m_i$ ) and articulation parameters ( $\phi$ ).
- The objective function minimizes the discrepancy between estimated relative transformations and observed 3D-3D correspondences, enforcing manifold constraints for joint axes and pivots.

C. Reconstruction & Global Alignment

TSDF Fusion: Two separate Truncated Signed Distance Function (TSDF) volumes are used:
1. Static Background: Integrated using camera poses, excluding dynamic regions (hands/moving parts).
2. Articulated Part: Reconstructed in its canonical coordinate frame by transforming depth maps using the inverse of the estimated part poses. This removes camera motion and occlusions, yielding a clean 3D model.
Global Alignment: Local submaps from all fragments (static and dynamic) are aligned using geometric correspondences (PREDATOR) and optimized via a scene-level pose graph to form a unified, globally consistent 3D scene.

D. Output

The final output is a mesh-based 3D scene where articulated parts can be manipulated along their inferred motion axes, with defined feasible ranges ( $\lambda_{min/max}$ or $\theta_{min/max}$ ).

3. Key Contributions

First End-to-End Solution: FunREC is the first method to reconstruct scene-scale, physically grounded digital twins directly from unstructured, in-the-wild egocentric interaction videos without CAD priors or multi-state captures.
Training-Free Pipeline: It leverages foundation models (VLM, SAM, TAPIP3D) for semantic and motion understanding, avoiding the need for task-specific training data.
New Datasets: The authors introduce two new benchmarks:
- RealFun4D: 351 real-world interaction videos from 60 apartments across 4 countries.
- OmniFun4D: 127 photorealistic simulated sequences in OmniGibson scenes.
Simulation Readiness: The method outputs URDF/USD files with inferred physical properties, enabling direct use in robotics simulators.

4. Experimental Results

FunREC was evaluated on HOI4D, RealFun4D, and OmniFun4D, outperforming state-of-the-art baselines (MonST3R, SpatialTrackerV2, BundleSDF, ArtGS) across all metrics:

Articulated Motion Estimation:
- Achieved ~5.3° axis direction error and 0.03m position error on OmniFun4D (vs. >30° and >0.9m for baselines).
- Reduced joint state error by 5-10× compared to previous methods.
Part Segmentation:
- Achieved 77.9% mIoU on OmniFun4D, a massive improvement over the next best (23.6% for MonST3R).
6D Pose Estimation:
- Achieved 79.0% ADD-S and 71.3% ADD on OmniFun4D, more than doubling the performance of BundleSDF.
Reconstruction Quality:
- Achieved 3.2 cm Chamfer Distance on OmniFun4D and 0.7 cm on HOI4D, significantly lower (better) than all baselines.
Robustness: FunREC maintained a 0% failure rate on RealFun4D and HOI4D, whereas other methods frequently failed to process or predicted incorrect joint types.

5. Significance and Applications

The work bridges the gap between passive 3D scene understanding and active, functional interaction. Its significance lies in:

Robotics & Embodied AI: Providing robots with "functional twins" that understand how to interact with objects (e.g., opening a fridge) rather than just where they are.
Simulation: Enabling the automatic generation of interactive simulation environments (URDF/USD) from real-world scans, crucial for training reinforcement learning agents.
Affordance Mapping: The system can infer hand-object contact regions and affordances, allowing for hand-guided interaction planning.
Real-World Applicability: By operating on casual, single-view videos, it removes the barrier of expensive multi-scan setups, making functional 3D reconstruction accessible for real-world deployment.

In summary, FunREC represents a paradigm shift from static reconstruction to functional reconstruction, enabling agents to perceive, plan, and act within dynamic, articulated 3D environments.