💻 computer science

MomaGraph: State-Aware Unified Scene Graphs with Vision-Language Model for Embodied Task Planning

The paper introduces MomaGraph, a unified, state-aware scene graph representation and a corresponding large-scale dataset and evaluation suite, alongside MomaGraph-R1, a vision-language model that leverages these resources to achieve state-of-the-art performance in task-oriented scene graph prediction and embodied task planning.

Original authors: Yuanchen Ju, Yongyuan Liang, Yen-Jen Wang, Nandiraju Gireesh, Yuanliang Ju, Seungjae Lee, Qiao Gu, Elvis Hsieh, Furong Huang, Koushil Sreenath

Published 2026-02-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuanchen Ju, Yongyuan Liang, Yen-Jen Wang, Nandiraju Gireesh, Yuanliang Ju, Seungjae Lee, Qiao Gu, Elvis Hsieh, Furong Huang, Koushil Sreenath

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a guest in a massive, high-tech mansion. You want to make a cup of tea, but you’ve never been there before. You see a kitchen, but you don't know which knob turns on the stove, which button starts the kettle, or if the "handle" you see is for a drawer or a cupboard.

To a standard robot, this mansion is just a giant, confusing pile of shapes and colors. To solve this, researchers have created MomaGraph, which essentially gives a robot a "mental map" that is much smarter than a simple GPS.

Here is the breakdown of how it works using everyday analogies:

1. The Problem: The "Blind Tourist" Robot

Most current robots are like tourists looking through a tiny straw. They can see an object (like a stove), but they don't understand its "personality."

Spatial knowledge is knowing where the stove is (it's in the corner).
Functional knowledge is knowing how it works (this knob controls the heat).

Current robots usually only have one or the other. They might know where the stove is, but they try to "open" it like a door because they don't understand it has knobs. They are like someone who knows where the library is but doesn't realize you have to open a book to read it.

2. The Solution: MomaGraph (The "Smart Blueprint")

MomaGraph is like giving the robot a "Smart Blueprint" of the room. Instead of just a map of walls, this blueprint has "sticky notes" on everything.

It connects the dots: It doesn't just say "there is a knob" and "there is a stove." It draws a line between them that says: "This specific knob is the boss of this specific burner."
It looks at the details: It doesn't just see a "fridge"; it sees the "handle" as a specific part you need to grab.
It’s Task-Focused: If you tell the robot "Make tea," it doesn't waste brainpower thinking about the sofa. It "zooms in" on the kettle, the water, and the stove, just like a human does.

3. The Brain: MomaGraph-R1 (The "Experienced Chef")

To make this blueprint, they built a brain called MomaGraph-R1. They trained it using something called Reinforcement Learning.

Think of this like training a puppy or a chef. Instead of just showing the robot a thousand pictures of kitchens (which is what most AI does), they gave it a "reward" every time it drew an accurate blueprint.

If the robot drew a map that correctly linked the remote to the TV, it got a "digital treat."
If it missed a step (like trying to boil water before plugging in the kettle), it didn't get a treat.
Over time, the robot stopped just "guessing" what things looked like and started "reasoning" about how they work.

4. The "Aha!" Moment: State-Awareness (The "Detective")

One of the coolest parts is that the robot is a detective.
Imagine a kitchen with four identical stove knobs. The robot doesn't know which one is the right one. But in MomaGraph, the robot can try one, see that the flame turns on, and say, "Aha! I've solved the mystery. This knob is the controller!" It updates its mental map in real-time as it interacts with the world. It learns from its own actions.

Summary: Why does this matter?

In short, MomaGraph moves robots from being "clumsy movers" (who can move from point A to point B) to "capable helpers" (who understand that to turn on the light, they must find the switch, not the lamp).

It gives them the ability to look at a messy, complicated room and say: "I see the goal, I see the tools, and I know exactly how they connect to get the job done."

Technical Summary: MomaGraph: State-Aware Unified Scene Graphs with VLM for Embodied Task Planning

MomaGraph is a research framework designed to enhance the ability of mobile manipulators to perform complex, long-horizon tasks in household environments. The work introduces a novel scene representation, a large-scale task-driven dataset, a specialized evaluation benchmark, and a reinforcement learning-trained Vision-Language Model (VLM).

1. The Problem: Limitations of Current Scene Representations

Existing scene graphs for embodied AI face three critical bottlenecks that hinder effective task planning:

Relationship Fragmentation: Most current graphs focus on either spatial relationships (where objects are) or functional relationships (how objects interact), but rarely both. This leads to incomplete representations that lack either geometric context or operability.
Static Nature: Existing methods often treat scenes as static snapshots, failing to account for state changes (e.g., a door being open vs. closed) or the dynamic updates required when an agent interacts with the environment.
Lack of Task Relevance: Standard scene graphs represent the entire environment indiscriminately. However, human perception is task-oriented; humans "zoom in" on specific objects and parts relevant to a current goal, whereas current models struggle to filter out irrelevant information.

2. Methodology: The MomaGraph Framework

The authors propose a multi-layered approach to solve these issues:

A. Unified Scene Representation (MomaGraph):
The core innovation is a task-oriented scene graph $G_T = (N_T, E^s_T, E^f_T)$ that integrates:

Nodes ( $N_T$ ): Includes both whole objects and fine-grained parts (e.g., knobs, handles, buttons) that are actionable.
Spatial Edges ( $E^s_T$ ): Encodes geometric relations (e.g., left of, touching).
Functional Edges ( $E^f_T$ ): Encodes interaction logic (e.g., control, activate, pair with).
State-Aware Updates: A dynamic update function $U(\cdot)$ allows the graph to evolve. If an agent interacts with an object and observes a state change, the graph prunes incorrect hypotheses and strengthens confirmed functional correspondences (e.g., identifying which specific knob controls a specific burner).

B. Graph-then-Plan Paradigm:
Instead of direct planning from raw pixels (which is prone to hallucination), the model follows a two-step process:

Graph Generation: The VLM constructs a structured, task-specific scene graph from multi-view images.
Task Planning: The VLM uses this structured graph as an intermediate reasoning step to generate a reliable action sequence.

C. MomaGraph-R1 & Reinforcement Learning:
The authors developed MomaGraph-R1, a 7B VLM based on Qwen2.5-VL. To optimize it, they used the DAPO reinforcement learning algorithm with a custom graph-alignment reward function. This reward incentivizes:

Correct action type prediction.
Semantic similarity in spatial-functional edges.
Node completeness (Intersection-over-Union of task-relevant objects).

3. Key Contributions

MomaGraph-Scenes Dataset: The first large-scale dataset featuring richly annotated, task-driven scene graphs with part-level interactive elements and multi-view observations.
MomaGraph-Bench: A systematic evaluation suite measuring six reasoning capabilities: Action Sequence, Spatial, Affordance, Precondition/Effect, Goal Decomposition, and Visual Correspondence.
MomaGraph-R1 Model: A 7B VLM that achieves state-of-the-art performance in open-source models by leveraging RL to master spatial-functional reasoning.

4. Results and Evaluation

Benchmark Performance: MomaGraph-R1 achieved 71.6% accuracy on MomaGraph-Bench, outperforming the best open-source baselines by 11.4%. It demonstrated performance competitive with closed-source models like GPT-5 and Claude-4.5-Sonnet.
Robustness to Complexity: While most models' performance drops sharply as task difficulty increases (from Tier 1 to Tier 4), MomaGraph-R1 maintains much higher stability, proving its capability for long-horizon planning.
Visual Correspondence: The model showed an emergent ability to maintain object consistency across different viewpoints, outperforming existing models on the BLINK benchmark.
Real-World Validation: On a RobotEra Q5 bimanual humanoid, the system successfully executed complex tasks (e.g., "Turn on the light closest to the remote and turn on the monitor") with a 70% overall success rate in unseen environments.

5. Significance

This work shifts the paradigm of embodied AI from "pixels-to-actions" toward "pixels-to-structured-reasoning-to-actions." By unifying spatial and functional knowledge into a dynamic, part-level graph, MomaGraph provides a more scalable and interpretable foundation for robots to operate in complex, unpredictable human households.