🤖 AI

MatterDoor: Sampling Zero-shot Spatio-semantic Priors using Generative Models

The paper introduces MatterDoor, a benchmark and pipeline that leverages off-the-shelf generative vision models to sample zero-shot spatio-semantic priors of occluded indoor spaces from doorway views, enabling robots to reason about hidden structures and objects without task-specific fine-tuning.

Original authors: Subhransu S. Bhattacharjee, Hao Lu, Dylan Campbell, Rahul Shome

Published 2026-06-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Subhransu S. Bhattacharjee, Hao Lu, Dylan Campbell, Rahul Shome

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a robot standing in a hallway, looking through a doorway into a room you've never seen before. You can see the floor right in front of you and maybe a bit of the wall, but the rest of the room is hidden behind the doorframe. Your mission? Find a specific object, like a "chair" or a "shelf," and walk over to it without bumping into anything.

The problem is, you can't see where the chair is, or even if there are obstacles blocking your path. You are essentially flying blind.

This paper, titled "MatterDoor," proposes a clever solution: instead of waiting to walk in and look around (which takes time and risks crashing), the robot uses a "creative imagination" to guess what the hidden room looks like before it moves.

Here is how they did it, explained simply:

1. The "Imagination Engine" (Generative AI)

The researchers used a type of AI called a generative model. Think of this AI as a very skilled artist who has seen thousands of photos of living rooms, kitchens, and offices.

The Input: You show the artist a small photo of the doorway.
The Prompt: You tell the artist, "I'm looking for a shelf."
The Magic: The artist doesn't just guess; they "paint" (generate) 100 different possible versions of what the rest of the room could look like. In some versions, the shelf is on the left; in others, it's on the right; in some, there's a big table blocking the way; in others, the path is clear.

2. Turning Paintings into Maps (The Pipeline)

The robot can't drive on a painting, so the system has to turn these images into 3D maps:

Step 1: The AI generates the hidden parts of the room (outpainting).
Step 2: Another AI looks at the picture and labels everything (e.g., "this is a wall," "this is a chair").
Step 3: A third AI estimates how far away things are (depth), turning the flat 2D pictures into 3D point clouds (like a digital cloud of dots representing the room).

Now, the robot has 100 different "what-if" scenarios of the room, all labeled and in 3D.

3. The "Safety Net" Strategy (Planning)

Instead of picking just one guess and hoping it's right (which is risky), the robot looks at all 100 guesses at once.

It asks: "In how many of these 100 worlds is the path to the chair clear?"
It asks: "In how many worlds is the chair actually there?"
It then plans a path that works well across most of these scenarios. If a path looks safe in 90 out of 100 guesses, it's a good path to take. If a path crashes into a wall in 50 of the guesses, the robot avoids it.

4. The Test: "MatterDoor"

To prove this works, the authors created a new test set called MatterDoor.

They took real 3D scans of rooms (from a dataset called Matterport3D).
They cut off the view so the robot only sees the doorway.
They asked the robot to find specific objects hidden in the room.
They compared their "imagination" method against robots that just guessed randomly or assumed the room was empty.

The Results

The paper found that:

Better Guesses: The AI-generated guesses were much closer to reality than random guessing. The robot could correctly guess where the object was about 90% of the time (within a meter).
Safer Driving: When the robot used these 100 guesses to plan its path, it crashed significantly less often than robots that only looked at the visible part of the room or assumed a single "best guess."
No Extra Training: The cool part is that they didn't have to teach the AI anything new. They just used "off-the-shelf" models that were already trained on general images and asked them to do this specific job.

The Bottom Line

The paper shows that robots can use powerful AI "imagination" tools to fill in the blanks of what they can't see. By generating many possible versions of a hidden room and planning for all of them, robots can navigate safely and find objects without needing to physically explore the space first.

What the paper does NOT claim:

It does not claim this works for real-world robots right now in a factory or home (they tested it in a simulation).
It does not claim the AI is perfect (sometimes the generated rooms had weird layouts, though rarely).
It does not claim this works for outdoor scenes or complex medical tasks. It is strictly about indoor rooms and finding objects.

Technical Summary: MatterDoor

Problem Statement

Autonomous robots operating in cluttered indoor environments frequently face heavy occlusion, particularly when navigating through doorways. From a partial egocentric RGB view, a robot may perceive the room type and local traversability but remains blind to critical geometric structures, free space, obstacles, and the locations of goal objects hidden within the room. Traditional planning approaches often rely on handcrafted, domain-specific priors or discrete abstractions that fail to support configuration-space planning with complex geometric constraints and semantic goals. Furthermore, existing methods struggle to derive a belief over unseen semantic structure without expensive active exploration or problem-specific fine-tuning.

The core problem addressed is how to derive zero-shot, offline spatio-semantic priors for robot reasoning using off-the-shelf pretrained generative vision models. The goal is to infer the likelihood of target objects and occupancy in hidden regions based solely on a partial doorway observation and a target query, enabling safe, goal-directed planning without access to ground-truth geometry during the planning phase.

Methodology

The proposed framework, MatterDoor, introduces a zero-shot pipeline that composes pretrained generative and discriminative models to generate labeled 3D point-cloud hypotheses of unobserved room interiors. The pipeline operates in four distinct stages:

VLM-Guided Gating: A Vision-Language Model (Qwen2.5-VL 7B) analyzes the doorway crop and the queried object category to score the plausibility of the target existing behind the doorway. This gate filters out unlikely queries (e.g., a "refrigerator" in a bathroom) to prevent wasted generation, while retaining a fallback mechanism for ground-truth-present targets that the VLM might miss.
Generative Outpainting: For accepted queries, a generative model (FLUX.1-Fill-dev) performs image outpainting. It expands the doorway crop by 500 pixels on the target side, conditioned on the visible doorway evidence and the target object label, to sample plausible hidden room completions. The model preserves the observed doorway geometry through a depth-based masking band.
Semantic Segmentation and Lifting: Each generated RGB completion is processed by a semantic segmentation model (ADE20K SegFormer-B5) to identify objects and free space. These 2D masks are then lifted into 3D using a metric monocular depth estimator (DepthPro).
3D Alignment and Hypothesis Formation: The lifted point clouds are affinely aligned to the estimated depth of the original crop using RANSAC to resolve scale ambiguities. The result is a set of $N$ (typically 100) semantically labeled 3D point clouds, representing an empirical belief over the hidden world.

These sampled worlds serve as the input for a sampling-based planner. The planner evaluates candidate trajectories against the ensemble of sampled worlds, calculating the probability of collision-free reachability to the target. The objective is to maximize the probability that a trajectory succeeds across the distribution of plausible completions, rather than optimizing for a single deterministic reconstruction.

Key Contributions

The paper makes three primary contributions:

A Zero-Shot Pipeline: A system that synthesizes labeled 3D point-cloud hypotheses from a single RGB doorway view without retraining or fine-tuning any components. The pipeline is directly queryable for collision, visibility, target evidence, and cost constraints.
MatterDoor Benchmark: A new benchmark derived from the Matterport3D dataset, consisting of 1,000 fixed doorway–object queries. It includes 100 hand-annotated doorway crops, ground-truth target presence labels, and room-level plausibility labels. The dataset covers five room types (bathroom, bedroom, kitchen, living room, office) and ten object classes.
Evaluation Protocol: A rigorous simulation-based evaluation framework using a Stretch mobile robot to measure safe object-reaching under uncertainty. The protocol evaluates both the quality of the generative priors (via localization and semantic metrics) and the utility of these priors for planning (via trajectory success and collision rates).

Results

The evaluation demonstrates that generative priors significantly outperform non-generative and deterministic baselines:

Generative Performance: The MatterDoor pipeline achieves a 90.3% target detection rate ( $p_{det}$ ) across samples, compared to 0% for "no generation" baselines. For object localization, it achieves a 90.1% mean Object Placement Recall (OPR) at 1m for a single sample, rising to 92.5% when considering the best sample in the budget. This vastly exceeds uniform random search (0.1% mean OPR) and deterministic centroid/frontier heuristics. The semantic distribution score ( $D_{sem}$ ) is also significantly lower (0.144) than a uniform object prior (0.642), indicating better alignment with real-world room-object statistics.
Planning Performance: When used by a sampling-based planner, the generative priors improve the target-reaching success rate from 6.9% (Uniform prior) to 23.1%. Crucially, the collision rate drops from 81.5% to 42.8%, and the fraction of collision-free trajectories increases from 31.6% to 88.4%.
Comparison to Deterministic Controls: A "1-sample" baseline (using only a single generated hypothesis) performs worse than the uniform prior, highlighting that a single inaccurate completion can mislead planning. In contrast, the ensemble approach (MatterDoor) leverages the diversity of samples to robustly estimate hidden obstacles and target locations.

Significance and Claims

The paper claims that pretrained generative vision models can serve as effective, planner-queryable belief sources for robotic reasoning under uncertainty, even without problem-specific fine-tuning.

Feasibility of Zero-Shot Priors: The results suggest that useful spatio-semantic priors for planning can be derived directly from general-purpose foundation models, challenging the notion that robotics requires specialized, handcrafted priors for occluded environments.
Value of Sampling over Determinism: The work demonstrates that sampling multiple plausible completions is superior to deterministic scene completion. While a single hypothesis may be geometrically plausible but semantically incorrect (leading to collisions), an empirical belief formed from multiple samples allows the planner to reason about the probability of success and failure, leading to safer trajectories.
Benchmarking Hidden-World Reasoning: MatterDoor provides a standardized way to evaluate how well generative models can infer hidden structure for robotics, moving beyond simple image completion metrics to task-oriented planning metrics.

The authors maintain a modest stance, acknowledging limitations such as the reliance on a single source dataset (Matterport3D), potential biases in pretrained components (e.g., occasional outpainting errors violating room layout), and the current lack of closed-loop real-world evaluation. They position the work as a step toward using generative models for offline planning, with future work needed to integrate these priors into interactive, closed-loop policies.