💻 computer science

FunFact: Building Probabilistic Functional 3D Scene Graphs via Factor-Graph Reasoning

This paper introduces FunFact, a framework that leverages factor-graph reasoning to construct probabilistic, open-vocabulary 3D functional scene graphs by jointly inferring object relations through LLM-derived priors and geometric constraints, thereby improving ambiguity resolution and confidence calibration compared to existing methods.

Original authors: Zhengyu Fu, René Zurbrügg, Kaixian Qu, Marc Pollefeys, Marco Hutter, Hermann Blum, Zuria Bauer

Published 2026-04-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhengyu Fu, René Zurbrügg, Kaixian Qu, Marc Pollefeys, Marco Hutter, Hermann Blum, Zuria Bauer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you walk into a room and see a remote control, a TV, a lamp, and a light switch. A standard computer vision system might tell you, "That's a remote, that's a TV, that's a lamp." It knows what things are and where they are.

But it doesn't know how they work together. It doesn't know that the remote controls the TV, or that the switch on the wall turns on the lamp, not the ceiling fan.

This is the problem the paper FunFact tries to solve. It's like teaching a robot to not just see the furniture, but to understand the "story" of how the room functions.

Here is a simple breakdown of how they did it, using some everyday analogies.

1. The Problem: The "Guessing Game"

In the real world, things are tricky.

The Ambiguity: You see a remote and a TV. You think the remote controls the TV. But maybe it controls the sound system instead? Maybe the TV is off, so you can't tell if the remote works.
The Old Way: Previous AI methods looked at pairs of objects one by one. "Does the remote control the TV?" Yes/No. "Does the switch control the lamp?" Yes/No. They made these guesses in isolation, like playing a game of "Guess Who?" without looking at the whole board. This led to a lot of confusion and wrong guesses.

2. The Solution: The "Detective Agency" (FunFact)

The authors built a system called FunFact. Instead of guessing one pair at a time, FunFact acts like a detective agency that looks at the entire crime scene at once to solve the mystery.

Here is how it works, step-by-step:

Step A: Building the 3D Map (The "Sketch Artist")

First, the system takes photos of the room and builds a 3D model. But it doesn't just see blobs; it identifies specific parts.

It sees a "stove" and its "knobs."
It sees a "lamp" and its "switch."
It uses smart AI (called Foundation Models) to label these things, even if they've never seen that exact brand of stove before.

Step B: The "Hypothesis Generator" (The "Creative Writer")

Once it has the map, it asks a super-smart AI (like a large language model) to brainstorm ideas.

AI says: "Okay, we have knobs and burners. Usually, knobs control burners. We have switches and lights. Switches usually turn on lights."
It creates a list of possible connections. "Maybe Knob A controls Burner 1? Maybe Knob A controls Burner 2?"

Step C: The "Logic Puzzle" (The Factor Graph)

This is the magic part. Instead of picking the single best guess, FunFact writes down all the possibilities and puts them into a giant logic puzzle called a Factor Graph.

Think of this like a group of detectives in a room:

The Proximity Rule: "Knobs are usually close to the burners they control." (If a knob is on the other side of the room, it's probably not controlling that burner).
The Cardinality Rule: "Usually, one knob controls exactly one burner." (It's rare for one knob to control three burners at once).
The Common Sense Rule: "Switches usually control lights, not toasters."

The system runs a calculation (called Belief Propagation) where all these rules talk to each other.

Detective 1 says: "I think Knob 1 controls Burner 1 because they are close."
Detective 2 says: "Wait, if Knob 1 controls Burner 1, then Knob 2 must control Burner 2, because of the 'one-to-one' rule."
Detective 3 says: "But Knob 2 is actually near Burner 3! So maybe Knob 1 was wrong."

By letting all the clues talk to each other, the system resolves the confusion. It doesn't just guess; it calculates the probability of every connection being true.

Step D: The "Confidence Score"

The best part is that FunFact doesn't just say "Yes" or "No." It gives a confidence score.

"I am 99% sure this switch turns on that light."
"I am only 50% sure this remote controls that TV; it might be the sound system."

This is crucial for robots. If a robot is 50% sure, it knows to double-check before it tries to turn something on.

3. The New Playground: FunThor

To test this, the researchers couldn't just use existing data because it was too messy. So, they built a new video game world called FunThor (based on the AI2-THOR simulator).

They created perfect, detailed maps of kitchens, living rooms, and bathrooms.
They labeled every single part (every button, every knob) and every single relationship (which knob turns which burner).
This gave them a "perfect answer key" to see if their detective agency was actually solving the puzzles correctly.

The Result

When they tested FunFact, it was much better than previous methods at:

Finding the right connections: It figured out which switch controlled which light even when they were far apart.
Knowing when it's unsure: It gave low confidence scores to ambiguous situations, whereas other methods just guessed confidently and got it wrong.

Summary

FunFact is like upgrading a robot's brain from a "list maker" to a "critical thinker."

Old Robots: "I see a switch. I see a light. I guess they are connected." (Often wrong).
FunFact: "I see a switch and a light. They are far apart, but there is a wire running along the wall. Also, there are two other switches nearby. Based on the layout and common sense, I am 95% sure this switch controls that light, and I am only 40% sure about the other one."

This allows robots to navigate and interact with our messy, complex homes much more safely and intelligently.

1. Problem Statement

Current 3D scene understanding methods primarily focus on spatial geometry and semantic identification (e.g., "what" and "where"). However, for robotics and interactive systems, understanding functionality (e.g., "how" objects interact, which switch controls which light) is critical.

Existing approaches to functional scene understanding suffer from two main limitations:

Isolated Pairwise Reasoning: Most methods evaluate functional relationships between object pairs in isolation. They fail to capture the scene-wide interdependence that humans use to resolve ambiguities (e.g., distinguishing which of two switches controls a specific lamp when both are equidistant).
Poor Calibration: Existing models often predict the most likely connection without modeling the distribution of all plausible options, leading to overconfident predictions on ambiguous relations and a lack of calibrated confidence scores necessary for safe robotic decision-making.

2. Methodology: FunFact

FunFact is a framework that constructs probabilistic, open-vocabulary functional 3D scene graphs from posed RGB-D images. It operates in two main stages:

A. Scene Reconstruction (Object- and Part-Centric)

FunFact first builds a consistent 3D representation of the scene, focusing on both whole objects and their functional parts.

VLM-Guided Detection: It uses a Vision-Language Model (VLM, specifically GPT-4.1) to propose functional objects (e.g., "stove") and their parts (e.g., "knob," "burner") along with coarse bounding boxes.
Grounding & Filtering: These proposals are grounded using GroundingDINO and SAM (Segment Anything Model). A filtering mechanism removes hallucinations (non-existent parts) and ensures spatial consistency (e.g., parts must overlap significantly with their parent object).
3D Fusion: Validated 2D detections are back-projected and fused across multiple views to create a unified 3D point cloud map containing object-part hierarchies.

B. Functional Scene-Graph Creation via Factor Graphs

This is the core innovation. Instead of predicting edges independently, FunFact formulates the problem as a dual factor graph to perform joint probabilistic inference.

Dual Graph Formulation:
- Variables: The candidate functional relations (edges in the scene graph) are treated as binary variables ( $x_i \in \{0, 1\}$ ) in the factor graph.
- Factors: Constraints are applied via two types of factors:
  1. LLM-Derived Priors: An LLM proposes semantically plausible relations (e.g., "knob controls burner") and their cardinality patterns (one-to-one vs. one-to-many).
  2. Geometric Priors:
    - Proximity Factors: Penalize connections between distant objects (unless the relation is known to be remote, like a remote control).
    - Cardinality Factors: Enforce structural constraints (e.g., a one-to-one constraint ensures a single knob controls only one burner, and vice versa).
Probabilistic Inference: The system uses Belief Propagation to infer the joint distribution over all functional edges. This allows the model to resolve local ambiguities by leveraging global context (e.g., if Switch A is assigned to Light 1, Switch B is probabilistically pushed toward Light 2).
Output: The result is a calibrated confidence score for every potential functional edge, allowing the system to discard low-confidence edges and retain high-confidence ones.

3. Key Contributions

FunFact Framework: A novel pipeline that integrates foundation models (VLMs/LLMs) with structured probabilistic reasoning (Factor Graphs) to reconstruct open-vocabulary functional 3D scene graphs.
FunThor Dataset: A new synthetic benchmark built on AI2-THOR. Unlike previous datasets, FunThor provides:
- Part-level geometry (detailed meshes of knobs, handles, etc.).
- Dense, rule-based functional annotations covering both object-object and object-part relations.
- 720 images across 12 scenes (Kitchen, Living Room, Bedroom, Bathroom).
Holistic Probabilistic Modeling: The introduction of a dual factor graph formulation that jointly reasons over all edges, significantly improving confidence calibration compared to independent pairwise reasoning.

4. Experimental Results

The authors evaluated FunFact on SceneFun3D, FunGraph3D, and the new FunThor dataset.

Mapping Performance: FunFact achieved the highest Recall@K across all categories (Objects, Interactive Elements, Overall Nodes) compared to baselines like OpenFunGraph and ConceptGraph. It demonstrated superior ability to detect fine-grained interactive elements.
Functional Relation Discovery:
- On FunThor, FunFact improved overall triplet recall by 39.0 percentage points (R@3) over OpenFunGraph.
- It significantly outperformed baselines in resolving ambiguous relations (e.g., multiple switches and lights) where pairwise methods failed.
Confidence Calibration:
- FunFact achieved a significantly lower Expected Calibration Error (ECE).
- For ambiguous classes (like light switches), the calibration error dropped from 0.51 (OpenFunGraph) to 0.07 (FunFact), demonstrating that the model's confidence scores accurately reflect the probability of correctness.
Ablation Studies: Removing the factor graph reasoning module led to a drop in precision (due to unfiltered ambiguous edges), while removing the hierarchical object-part representation significantly hurt the detection of small interactive elements.

5. Significance and Impact

Robotic Reasoning: By providing calibrated confidence scores, FunFact enables robots to make safer decisions. A robot can identify when a functional relationship is uncertain and seek further verification (e.g., physical interaction) rather than acting on a potentially wrong assumption.
Beyond Static Pairs: The work shifts the paradigm from isolated pairwise classification to holistic scene reasoning, mimicking how humans use context to disambiguate functional relationships.
Benchmarking: The introduction of FunThor addresses the scarcity of dense, part-level functional annotations, setting a new standard for evaluating functional scene understanding in synthetic and potentially real-world environments.

In conclusion, FunFact demonstrates that combining semantic priors from Large Language Models with geometric constraints in a probabilistic factor graph framework yields a robust, calibrated, and highly accurate system for understanding the functional dynamics of 3D environments.