💻 computer science

Generating Humanless Environment Walkthroughs from Egocentric Walking Tour Videos

This paper presents a generative approach that utilizes a semi-synthetic dataset and a fine-tuned video diffusion model to realistically remove humans and their shadows from egocentric walking tour videos, thereby enabling the creation of high-quality humanless 3D/4D environment models.

Original authors: Yujin Ham, Junho Kim, Vivek Boominathan, Guha Balakrishnan

Published 2026-04-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yujin Ham, Junho Kim, Vivek Boominathan, Guha Balakrishnan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are holding a camera and walking through a busy city square in Tokyo, a bustling market in Marrakech, or a quiet street in Paris. You want to save this video to create a perfect, digital 3D model of that place later—maybe for a video game, a robot to navigate, or a virtual tour.

But there's a problem: The video is full of people.

Every time you walk, strangers cross your path, blocking your view of the buildings, the trees, and the ground. If you try to build a 3D model from this messy footage, the computer gets confused. It thinks the people are part of the building, or it gets lost because the view keeps changing. You can't get a clean picture of the "skeleton" of the city.

This paper introduces a solution called CrowdEraser. Think of it as a magical "Photoshop for video" that doesn't just cut people out, but rebuilds the world behind them so perfectly that you'd never know anyone was ever there.

Here is how they did it, broken down into simple steps:

1. The Problem: The "Crowded Room" Analogy

Imagine trying to take a photo of a beautiful painting on a wall, but a hundred people are standing in front of it, waving their arms. If you try to edit the photo to remove them, you might just leave a blank white hole or a blurry mess.

Existing AI tools (like the ones used before this study) are good at removing a single person standing far away. But when the camera is close to the ground (like a person walking) and the street is packed with a crowd, those tools fail. They leave weird shadows, blurry blobs, or they accidentally "hallucinate" (make up) fake people or objects in the empty space.

2. The Solution: The "Frankenstein" Dataset

To teach a computer how to fix this, you usually need a "Before" and "After" video. You need a video of the street with people, and the exact same video of the street without people.

The Catch: You can't film the same street twice. You can't get 100 people to freeze in place, walk away, and then film the empty street with the exact same camera movement and lighting. It's impossible.

The Clever Fix: The researchers built a Semi-Synthetic Dataset (let's call it "EgoCrowds").

The Background: They found hundreds of videos of empty streets (or streets with very few people) from all over the world.
The Foreground: They found videos of crowds walking.
The Magic Mix: They used a computer to "cut and paste" the walking people onto the empty backgrounds.
The Shadow Trick: Just pasting a person looks fake because they don't cast a shadow. The researchers wrote a special script to simulate realistic shadows that move and stretch just like real sunlight does.

Now, the computer has a perfect teacher: It sees the "fake" crowded video and knows exactly what the "real" empty background should look like underneath.

3. The Training: Teaching the AI to "Un-See" People

They took a powerful AI model (called Casper, which is like a very smart video editor) and trained it on this "Frankenstein" dataset.

The Lesson: "Here is a video with a person and a shadow. Now, tell me what the video looks like if you erase them."
The Result: The AI learned not just to delete the person, but to reconstruct the world behind them. It learned that if a person blocks a brick wall, the wall continues behind them. If a person casts a shadow on the pavement, the pavement is still there once the shadow is gone.

4. The Magic Tool: CrowdEraser

The final product is a tool called CrowdEraser.

Input: You give it a messy walking tour video and a mask (a digital outline) of where the people are.
Process: The AI looks at the movement of the camera and the surrounding pixels to guess what's hidden.
Output: It spits out a brand new video where the people and their shadows are gone, replaced by a seamless, realistic version of the empty street.

5. Why This Matters: The "Ghost City"

The researchers tested this by taking their new "human-free" videos and feeding them into a 3D reconstruction system.

Before: When they used the original crowded videos, the 3D model was broken, shaky, and full of holes because the computer couldn't agree on what the buildings looked like.
After: When they used the CrowdEraser videos, the 3D model was solid, detailed, and stable. It was as if they had walked through a "Ghost City" where everyone vanished, leaving only the architecture behind.

Summary

Think of CrowdEraser as a time machine for video. It takes a chaotic, crowded moment in time and "rewinds" the people out of existence, filling in the gaps with the perfect reality that was hidden underneath. This allows us to turn messy, real-world YouTube walking tours into clean, usable blueprints for robots, video games, and virtual worlds.

1. Problem Statement

Egocentric "walking tour" videos (first-person perspective) are a vast, underutilized resource for training visual models of urban environments. However, their direct application in static environment modeling (e.g., 3D reconstruction, robotics, digital twins) is severely hindered by the significant presence of transient humans.

The Challenge: In crowded scenes or when the camera wearer is close to others, humans and their associated effects (shadows, accessories) occlude a large fraction of pixels per frame.
Limitations of Existing Methods: State-of-the-art video inpainting and object removal models (specifically Casper, a diffusion-based model from GenOmnimatte) perform reasonably well on sparse crowds but fail significantly when faced with high-density crowds and complex outdoor backgrounds. They often produce artifacts, hallucinate objects within masked regions, or fail to remove shadows correctly.
Root Cause: The performance gap is attributed to a domain shift between the training data of existing models and the specific characteristics of walking tour videos, rather than a fundamental flaw in the diffusion architecture itself.

2. Methodology

The authors propose CrowdEraser, a generative framework designed to remove humans and their shadows from egocentric videos. The methodology consists of three core components:

A. Dataset Construction: EgoCrowds

Since capturing real video pairs of the same scene with and without crowds under identical lighting and camera motion is practically impossible, the authors constructed EgoCrowds, a semi-synthetic dataset of 1,000 video clip pairs.

Source Material: Real walking tour videos from 50 cities worldwide.
Background Extraction: Clips are filtered to ensure minimal human presence (<5 people in >90% of frames) using Grounded-SAM-2.
Foreground Extraction: Clips with varying crowd densities (0–50% of frame area covered by humans) are extracted and categorized into bins.
Composite Generation:
- Foreground humans are overlaid onto empty backgrounds.
- Shadow Simulation: A critical innovation. The system generates realistic shadows by applying affine transforms (rotation, shear, scaling) to human masks and convolving them with a Gaussian kernel. Shadow intensity and direction are randomized to simulate varying lighting conditions.
- Training Tuple: Each training instance consists of (Composite Video, Human Mask, Clean Background).

B. Model Architecture: CrowdEraser

The model is a fine-tuned version of Casper (a video diffusion model based on CogVideoX).

Training Strategy: The pre-trained Casper model is fine-tuned on the EgoCrowds dataset. The VAE and text encoder are frozen; only the 3D transformer layers are updated.
Loss Function: A composite loss is used to ensure both spatial accuracy and temporal consistency:
- $L_{base}$ : Standard diffusion denoising loss.
- $L_{sub}$ : Motion loss that constrains the temporal differences of predicted noise residuals between consecutive frames, preventing flickering and ensuring smooth dynamics.
Inference: The model takes an input video, a corresponding mask of the humans, and a text prompt ("A video of a beautiful empty, human-free scene") to generate the cleaned background.

C. Downstream Application

The authors demonstrate the utility of the generated humanless videos by feeding them into SpatialTrackerV2, a state-of-the-art 4D reconstruction model, to build 3D point clouds and scene geometry.

3. Key Contributions

EgoCrowds Dataset: The first large-scale dataset of paired walking tour videos (crowded vs. empty) with simulated shadows, enabling supervised training for this specific domain.
CrowdEraser Model: A specialized diffusion-based inpainting model that outperforms existing baselines in removing humans and their shadows from complex, crowded egocentric scenes.
Shadow-Aware Training: Demonstrated that explicitly simulating and training on shadow effects is crucial for realistic removal, preventing "floating" shadows in the output.
3D Reconstruction Enablement: Proved that removing transient humans is a necessary preprocessing step to enable stable and dense 3D/4D reconstruction of urban environments from raw walking tour footage.

4. Results

The authors evaluated CrowdEraser against three baselines: Casper, ProPainter, and DiffuEraser.

Quantitative Performance:
- CrowdEraser achieved the best PSNR (Peak Signal-to-Noise Ratio) and DreamSim (perceptual similarity) across all crowd density levels (0–50%).
- Performance gains were most significant as Crowd% (percentage of pixels occluded by humans) increased. Baselines degraded rapidly in high-density scenarios.
Qualitative Performance:
- Shadow Removal: Unlike ProPainter and DiffuEraser (which often blur or leave floating shadows) and Casper (which hallucinates objects in large masked areas), CrowdEraser successfully removed both humans and their cast shadows while preserving background texture.
- Temporal Consistency: The motion loss function ensured smooth video generation without flickering, even in long sequences with sustained occlusions.
Ablation Studies:
- Removing shadow simulation from training caused the model to fail at associating shadows with humans.
- Training only on low-density masks resulted in failure to inpaint large occluded regions (color bleeding).
3D Reconstruction:
- Raw videos with humans resulted in sparse, unstable 3D point clouds due to occlusion.
- Videos processed by CrowdEraser yielded denser, more consistent point clouds with stable tracking of static background features (e.g., walls, pavement patterns).

5. Significance and Limitations

Significance:

Unlocking Data: This work unlocks the potential of the massive corpus of YouTube walking tour videos for computer vision tasks that require static, unobstructed scene understanding.
Domain Adaptation: It highlights the importance of domain-specific dataset construction (semi-synthetic data with realistic effects) over relying solely on general-purpose foundation models.
Robotics & Simulation: The ability to generate clean, human-free walkthroughs is critical for training autonomous agents and creating digital twins of cities.

Limitations:

Generalization: The model may struggle with scenes visually distinct from the training distribution (e.g., extreme weather, non-urban environments).
Long-Range Consistency: For very long videos with sustained large occlusions, temporal consistency can degrade because the diffusion backbone processes fixed-length clips (85 frames).
Fine Details: In extremely high-density crowds, the model may struggle to reconstruct high-frequency fine-grained details, resulting in slight blurring.

Conclusion:
CrowdEraser provides a robust, practical solution for transforming raw, crowded egocentric footage into clean environmental walkthroughs, bridging the gap between available video data and the requirements for high-fidelity 3D scene modeling.