💻 computer science

InHabit: Leveraging Image Foundation Models for Scalable 3D Human Placement

The paper introduces InHabit, a fully automatic and scalable framework that leverages 2D image foundation models to generate a large-scale, photorealistic 3D human-scene interaction dataset, thereby addressing the scarcity of real-world data and significantly improving 3D human reconstruction and contact estimation tasks.

Original authors: Nikita Kister, Pradyumna YM, István Sárándi, Jiayi Wang, Anna Khoreva, Gerard Pons-Moll

Published 2026-04-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nikita Kister, Pradyumna YM, István Sárándi, Jiayi Wang, Anna Khoreva, Gerard Pons-Moll

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to live in a house. You can show it blueprints of the house (the 3D geometry), and you can show it videos of people dancing in a studio (motion capture). But you have a huge problem: you don't have enough videos of people actually living in that specific house.

You don't have footage of someone leaning on the kitchen counter, sitting on the sofa to watch TV, or crouching to pick up a toy in the hallway. Real-life filming is expensive, slow, and hard to do in every possible room.

Enter InHabit. Think of it as a magical, automated interior designer and director that can instantly fill any empty house with realistic people doing the right things, without a single human actor or camera crew.

Here is how it works, broken down into three simple steps:

1. The "Brain" (The Vision-Language Model)

First, the system looks at a 3D model of an empty room (like a digital twin of a house). It asks an AI "brain" (a Vision-Language Model) a simple question: "What would a human naturally do in this room?"

Old way: You might tell a computer, "Put a person here." The computer would just stand a person up in the middle of the room, looking awkward.
InHabit way: The AI "brain" looks at the room and thinks, "Oh, that's a kitchen with a stove. A person would probably be cooking. That's a sofa with a TV? Someone would be watching a show." It uses its knowledge of the real world to come up with a script for the scene.

2. The "Artist" (The Image-Editing Model)

Once the "brain" decides what action to take (e.g., "cooking"), it passes the order to an AI "artist" (an Image-Editing Model). This artist is like a master painter who has seen millions of photos of people cooking.

The artist doesn't just paste a sticker of a person onto the picture. Instead, they paint a new person directly into the scene. They make sure the person is holding a spoon, leaning toward the stove, and looking at the pot. The pose is perfect because the artist has "seen" this action a million times in photos on the internet.

3. The "Architect" (The 3D Lifting)

Now, we have a beautiful 2D picture with a person in it, but we need them to exist in the 3D world so a robot can interact with them. This is the tricky part.

Imagine taking a flat drawing of a person and trying to turn it into a 3D statue that fits perfectly into the room's furniture. InHabit uses a mathematical "architect" to do this. It takes the 2D painting and sculpts a 3D body (using a standard human shape called SMPL-X) that fits exactly where the painting put them.

It checks for physics:

Is the person floating? (No, the feet are on the floor).
Are they walking through the wall? (No, the body stops at the wall).
Is the size right? (Yes, they aren't a giant or a tiny person).

Why is this a Big Deal?

1. It's a "Copy-Paste" Machine for Reality
Before this, if you wanted to train a robot to understand how humans interact with objects, you needed expensive motion-capture suits and actors. InHabit can generate 78,000 different scenarios across 800 different virtual houses automatically. It's like having a factory that prints out "human behavior" instead of plastic toys.

2. It Understands "Common Sense"
Old methods were like a robot that only knows geometry: "If there is a flat surface, put a person on it." This leads to weird results, like a person standing on a ceiling or sitting on a hanging lamp. InHabit understands context. It knows you sit on a chair, not under it, and you cook at a stove, not inside the fridge.

3. It Makes Robots Smarter
The authors tested this new data by teaching robots to recognize where humans are touching things. When the robots were trained on InHabit's data, they got much better at guessing human behavior than when trained on old data. In fact, in a test where humans voted on which robot behavior looked most real, 78% of people preferred InHabit's creations over the best existing methods.

The Bottom Line

InHabit is a tool that takes the "common sense" knowledge of 2D internet images and turns it into 3D reality. It solves the "data shortage" problem by automatically generating millions of realistic scenarios where humans interact with their environment, helping us build better robots and AI that can truly understand how we live.

1. Problem Statement

The core challenge addressed is the scarcity of large-scale, semantically meaningful 3D human–scene interaction (HSI) data.

Limitations of Real Data: Capturing real-world interactions via motion capture (MoCap) is logistically difficult, expensive, and limited to controlled studio environments or small numbers of scenes (typically <30 scenes).
Limitations of Existing Synthetic Data: Current synthetic approaches rely on geometric heuristics (collision avoidance, contact optimization) that lack semantic context. They can place a human on a surface but cannot reason why (e.g., cooking at a stove vs. standing on a chair to change a lightbulb).
The Gap: There is a need for a scalable method to generate diverse, context-aware 3D human interactions that leverage the commonsense knowledge of how humans interact with environments, which is currently only implicit in 2D foundation models trained on internet-scale data.

2. Methodology: The "Render–Generate–Lift" Pipeline

The authors propose InHabit, a fully automated, scalable data generation engine that transfers 2D foundation model knowledge into the 3D domain. The process follows a four-stage pipeline:

Stage 1: Scene-Aware View Selection & Rendering

Input: Static 3D indoor scenes (e.g., from the Habitat-Matterport 3D dataset).
Process: The system automatically samples camera viewpoints. A depth-based heuristic filters views to ensure the floor is visible (for grounding) and the view extends deep enough into the room to provide rich context.

Stage 2: Affordance-Aware 2D Human Insertion

This stage populates rendered 2D views with humans using two foundation models:

Vision-Language Model (VLM) for Reasoning: The VLM analyzes the rendered scene to propose contextually meaningful actions. It uses two strategies:
- Affordance-driven: Identifying what the scene allows (e.g., "kitchen" $\to$ "cooking").
- Action-driven: Contextualizing generic actions (e.g., "standing" in a room with a high light $\to$ "standing on a chair").
- Constraint: The VLM is constrained to propose interactions that do not alter the scene geometry (no moving furniture).
Image-Editing Model for Synthesis: An image-editing foundation model takes the VLM's text prompt and the scene image to generate photorealistic humans performing the action. Crucially, this model adapts body poses, proportions, and spatial placement to the local scene context (e.g., facing a TV, reaching for a shelf).

Stage 3: 3D Scene-Aware Lifting

The 2D generated image is lifted back into 3D to recover a physically plausible SMPL-X human mesh aligned with the known scene geometry.

Initialization: A monocular depth estimator recovers a partial human point cloud from the edited image, which is back-projected and aligned with the scene mesh.
Optimization: The system optimizes human parameters (pose $\theta$ $θ$ , shape $\beta$ $β$ , translation $t$ $t$ , and a new scale parameter $s$ ) using a multi-term objective function:
$\min \lambda_{proj} \mathcal{L}_{proj} + \lambda_{depth} \mathcal{L}_{depth} + \lambda_{con} \mathcal{L}_{contact} + \lambda_{pen} \mathcal{L}_{penetration} + \lambda_{reg} \mathcal{L}_{reg}$
- Key Innovation: The explicit scale parameter $s$ decouples metric sizing from body shape, allowing the optimizer to match the scene scale without distorting the body silhouette.
- Constraints: The method leverages the complete 3D scene mesh (unlike previous methods that use partial depth) to enforce stable contact and penetration constraints, ensuring the human is physically grounded.

Stage 4: Post-Reconstruction Filtering

To ensure high quality, three filters remove physically implausible artifacts:

Visibility Filter: Removes humans placed in mirrors/glass or severely truncated figures by checking point cloud visibility against the segmentation mask.
Depth-Based Boundary Filter: Detects hallucinated geometry (e.g., displaced furniture) near the human by comparing the edited image's depth map against the original scene depth.
Volume Filter: Discards meshes with anomalously low volume (e.g., fitting an adult model to a child or a poster), which indicates a failure in the lifting process.

3. Key Contributions

InHabit Method: A fully automated, scalable pipeline combining VLM-based affordance reasoning, generative image editing, and optimization-based 3D reconstruction to create 3D HSI data without manual annotation.
InHabitants Dataset: The first large-scale photorealistic 3D HSI dataset containing 78,000 samples across ~800 building-scale scenes. It includes complete 3D geometry, SMPL-X body meshes, and RGB images, covering diverse single and multi-person interactions.
Performance Validation: Demonstrated that synthetic data generated by InHabit outperforms existing SOTA placement methods (GenZI, POSA) in both quantitative metrics and perceptual user studies.

4. Experimental Results

The authors evaluated InHabit through quantitative benchmarks, qualitative comparisons, and user studies.

Qualitative & User Study:
- In a perceptual study with 726 responses, 78.1% of participants preferred InHabit's interactions over GenZI (6.7%) and POSA++ (15.2%).
- InHabit produced more natural, context-aware poses (e.g., touching a statue, leaning on a counter) compared to the geometrically valid but semantically empty poses of baselines.
Contact Estimation (DAMON Benchmark):
- Augmenting standard training data with InHabitants improved the SOTA contact estimator DECO.
- F1-score increased by 7.4% and Recall by 10.6% compared to training on DAMON alone.
- Notably, InHabitants (automatically generated) outperformed training with TRUMANS (labor-intensive real-world MoCap data), attributed to InHabit's superior scene diversity (~800 scenes vs. 100).
3D Human-Scene Reconstruction (PROX & RICH):
- Training feed-forward reconstruction models (Human3r and GRAFT) on InHabitants data yielded significant improvements over models trained on original datasets (BEDLAM, PROX).
- GRAFT trained exclusively on InHabitants generalized to the outdoor RICH benchmark, despite never seeing outdoor images during training, highlighting the robustness of the learned interaction priors.

5. Significance and Impact

Bridging 2D and 3D: The paper successfully demonstrates that internet-scale 2D foundation models encode sufficient commonsense knowledge of human-environment interactions to serve as a practical, scalable alternative to expensive 3D motion capture.
Scalability for Embodied AI: By solving the data scarcity bottleneck, InHabit enables the training of embodied agents that can understand and interact with 3D scenes in a human-centric way (e.g., robotics, virtual assistants).
Open Source: The authors commit to releasing the InHabitants dataset and code, fostering further research in 3D scene understanding and human-robot interaction.

In summary, InHabit represents a paradigm shift from geometric-heuristic-based placement to semantic-aware generation, leveraging the power of modern foundation models to create the first truly large-scale, diverse, and contextually rich 3D human interaction dataset.