💻 computer science

Beyond Placement and Articulation: Usage-Driven Code Scenes for Embodied Interaction

This paper introduces RoomWright, an agentic framework that generates executable, code-based 3D indoor scenes for embodied AI by shifting focus from mere visual placement to usage-driven object reasoning and rule-based interaction modeling.

Original authors: Zijian Xiao, Zipeng Ye, Jinkun Hao, Xiong Yang, Yuchen Xie, Ran Yi

Published 2026-08-20

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Zijian Xiao, Zipeng Ye, Jinkun Hao, Xiong Yang, Yuchen Xie, Ran Yi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where a robot can walk into a room, pick up a kettle, and place it on a stove, or where a virtual agent can flip a switch to turn on a lamp. For these machines to learn and act, they need environments that are not just pictures of rooms, but places that actually work. They need spaces where objects have parts that move, where a door handle turns a latch, and where pressing a button changes the state of a light. Until recently, the digital rooms built for these robots were mostly static. They looked real, but they did not behave real. A robot could bump into a chair, but it could not pull a drawer open or understand that a fan's speed changes when a dial is turned. The challenge has been to build indoor spaces that are not just visual collections of furniture, but functional stages where cause and effect play out exactly as they do in the physical world.

A team of researchers has introduced a new way to build these interactive worlds, moving beyond simply placing objects in a scene to understanding how those objects are actually used. They call their system RoomWright. Instead of creating a room by arranging 3D models like pieces on a board, the system writes the entire room as a set of instructions, or code, that a computer can run. This approach allows the digital environment to be fully editable and ready for simulation. The key insight is that a room is defined by the activities that happen inside it. If you are making a coffee, you need a counter, a kettle, a power source, and a way to turn it on. The researchers found that by starting with the activity and working backward to the objects and their connections, they could generate scenes that are not only visually correct but also functionally logical.

Previous methods often relied on looking at images of rooms to guess where things should go. While this works for making things look pretty, it often fails to capture the hidden logic of how things work together. A robot might see a kettle and a stove, but without understanding the function, it might place the kettle floating in the air or facing the wrong way. The new system solves this by treating every major piece of furniture as a center for a specific task. It asks, "What does a person need to do here?" and then brings in only the objects required for that task. If the task is cooking, the system adds a stove, a counter, and a kettle, and it ensures they are placed where a human could actually reach them. This process, which the researchers call usage-driven reasoning, ensures that the room is built around human needs rather than just visual appeal.

The system goes a step further by teaching the objects how to interact with each other. In a real kitchen, turning a knob on a fan changes the speed of the blades, and pressing a button on a microwave lights up the display. These are not just movements of a single part; they are chains of events where one action triggers another. The researchers programmed their system to write rules for these interactions. Each rule acts like a small script: if a user presses a switch, then the lamp turns on. If a user rotates a dial, then the fan speed changes. By writing these rules as code, the system can create complex behaviors where multiple objects work together, such as a kettle sitting on a base that activates a light when the kettle is placed correctly. This allows the digital room to react to a robot's actions in a way that feels natural and consistent.

One of the trickiest parts of building these rooms is figuring out which way an object should face. A toaster has a front, a back, and a top, but a computer looking at a picture might not know which side is the front unless it sees a person using it. The researchers solved this by using the information they already had while building the object. Because the system constructs the room from code, it knows the details of every part, like the location of a handle or a button. It uses this knowledge to decide the best orientation for the object based on how a human would use it. This ensures that a keyboard is placed with the keys facing the user and a mouse is positioned for a right-handed grip, something that previous methods often got wrong because they relied only on visual guesses.

The team tested their system by creating twenty-eight different rooms, ranging from dining areas to offices, and compared the results against other existing methods. The tests showed that their approach produced rooms that were far more complete and realistic. In their simulations, the generated rooms had fewer objects floating in mid-air, better support for furniture, and more objects that were actually reachable and usable. Most importantly, the rooms contained a much higher number of interactive elements. While other methods might create a room with a few moving parts, the new system generated scenes with many objects that could be manipulated, such as doors that open, lights that switch on, and dials that control speed. The researchers demonstrated that a robot could enter these digital rooms and successfully perform tasks like grasping a door handle to close it or pressing a button to light up a display.

This work represents a significant shift in how we build digital environments for robots and artificial intelligence. By focusing on the purpose of a room rather than just its appearance, the researchers have created a way to generate spaces that are ready for real-world testing. The resulting scenes are not just static images but dynamic environments where every object has a role, and every action has a consequence. This allows robots to learn and practice their skills in a world that behaves exactly like the one we live in, bridging the gap between digital simulation and physical reality. The system proves that when we build digital worlds with the same logic we use in our own homes, we create a much better foundation for the machines that will one day live and work alongside us.

Technical Summary: RoomWright

Problem Statement

Indoor scene synthesis is critical for embodied AI, robotic manipulation, and simulation-based policy learning. While recent code-based methods (e.g., SceneCode, Code-as-Room) have advanced the generation of editable, extensible 3D environments, they remain largely focused on visual construction and object-level articulation (e.g., doors opening, drawers sliding). These approaches often fail to model the functional usage of scenes. Consequently, generated scenes lack the causal dependencies and multi-object interactions required for realistic embodied tasks (e.g., pressing a switch to turn on a lamp, or placing a kettle on a base to activate an indicator). Furthermore, determining the correct orientation of manipulable objects from visual pixels alone is ambiguous, leading to unrealistic placements that hinder interaction.

Methodology: RoomWright

RoomWright is an agentic framework that generates 3D scenes represented entirely as executable code, designed specifically for embodied interaction. The system operates through three primary stages:

1. Scene Planning (Usage-Driven Object Reasoning)

Given a natural language room description, the system constructs a symbolic scene state $S = (O, F, G_x)$ :

Object Set ( $O$ ): Instead of sequential enumeration, the system performs usage-driven object reasoning. It treats major floor-standing anchors as "task centers." For each anchor, it identifies the human activity it supports and radially admits only the companion objects necessary for that task. This ensures the scene composition is grounded in intended usage rather than visual plausibility alone.
Support Forest ( $F$ ): Records the physical support relationships (e.g., what rests on what) to ensure geometric stability.
Interaction Graph ( $G_x$ ): Exhaustively induces interaction edges $e = (s, t, m, \phi)$ for every object, linking controllers (sources) to targets via specific methods, capturing affordances that users can actuate.

2. Simulation-Ready Asset Generation

The system generates assets that are physically executable:

Articulated Code: Using a backend (Articraft), objects are compiled into simulation-ready assets with semantic parts, collision geometry, and dynamic properties (mass, inertia, friction). Hinged and sliding parts are converted into revolute and prismatic joints.
Code-Based Multi-Part Interaction: A code agent compiles interaction edges into trigger–condition–effect rules over structured object states. Unlike isolated articulation parameters, these rules capture causal dependencies across multiple objects and parts (e.g., a switch triggering a lamp state change). The runtime binds abstract part names to concrete, addressable links in the simulation.
Material Generation: Textures and PBR materials are generated using FLUX.2 and applied via closed-form UV projections to ensure consistent appearance without explicit UV unwrapping.

3. Layout Generation

Geometric Solving: A symbolic layout program, expressed in a user-centric reference frame, is grounded by a geometric solver into collision-free metric poses.
Usage-Guided Orientation: To resolve the ambiguity of manipuland orientation, the system leverages annotation-informed usage-guided orientation. It fuses semantic annotations (recorded during modeling) with rendered views. A Vision-Language Model (VLM) predicts a "use-facing" direction based on inferred hand and arm anchors, ensuring objects like keyboards and mice are oriented correctly for interaction.
Iterative Refinement: A vision-in-the-loop critic evaluates visual realism, functional consistency, and physical validity, refining the layout through iterative scene editing and physics simulation validation.

Key Contributions

RoomWright Framework: A code scene generation agentic framework that produces editable, simulation-ready environments for embodied interaction, moving beyond static geometry to functional scenes.
Usage-Driven Object Reasoning: A novel approach to scene composition that derives object selection and spatial organization from intended activities and functional relationships, rather than visual judgment alone.
Multi-Part Interaction via Code: The introduction of a code agent that represents interactions as trigger–condition–effect rules, enabling coordinated motion and causal dependencies across multiple objects and parts.
Annotation-Informed Orientation: A method to alleviate orientation ambiguity for manipulable objects by using semantic annotations from the modeling stage to guide the assignment of use-facing directions.

Results

Extensive experiments were conducted on 28 diverse room-level prompts, comparing RoomWright against SceneCode, Code-as-Room, and SceneSmith.

Quantitative Performance: RoomWright achieved the highest scores in Access (reachability of functional sides, 65.6% vs. 39.8% for SceneSmith) and ObjAttr (object attribute satisfaction, 64.6% vs. 42.5%). It also maintained high scores in collision-free placement (95.9%) and navigability (97.3%).
Interactive Content: RoomWright significantly outperformed baselines in generating interactive objects (16.4 vs. 4.4 for SceneCode) and complex non-hinge interactions (CHI: 6.4 vs. 0 for others). It was the only method to support Multi-Part Coordination (MPC).
Qualitative & Ablation: Visual comparisons showed RoomWright produced layouts with higher plausibility and realistic visual quality. Ablation studies confirmed that removing usage-driven reasoning led to missing contextually necessary objects, and removing usage-guided orientation resulted in incorrect placement of manipulable items (e.g., keyboards and mice).
Robot Integration: The generated scenes were successfully imported into NVIDIA Isaac Sim (via OmniGibson), where robots could successfully execute tasks such as grasping handles, pressing buttons, and observing state changes (e.g., lights turning on).

Significance and Claims

The paper claims that RoomWright represents a shift from visual construction to functional modeling. By treating scenes as executable code that encodes not just geometry but also usage logic, causal dependencies, and interaction states, RoomWright provides environments that are "executable, editable, and simulation-ready." The authors argue that this approach is essential for embodied AI, as it allows agents to interact with environments that behave according to physical laws and functional intent, rather than merely looking realistic. The work demonstrates that code-based representations can effectively capture the "how a scene is actually used," bridging the gap between procedural generation and interactive simulation.