💻 computer science

STABLE: Simulation-Ready Tabletop Layout Generation via a Semantics-Physics Dual System

STABLE is a novel dual-system framework that combines a fine-tuned LLM for semantic layout generation with a physics-aware flow-based model for physical correction, enabling the progressive creation of simulation-ready tabletop scenes that strictly adhere to task instructions while ensuring collision-free and physically plausible object arrangements.

Original authors: Zhen Luo, Yixuan Yang, Xudong Xu, Jinkun Hao, Zhaoyang Lyu, Feng Zheng, Jiangmiao Pang, Yanwei Fu

Published 2026-05-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhen Luo, Yixuan Yang, Xudong Xu, Jinkun Hao, Zhaoyang Lyu, Feng Zheng, Jiangmiao Pang, Yanwei Fu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a robot chef tasked with setting a table for a complex dinner. You need to place a knife, a fork, a plate, and a glass exactly where the recipe (the instruction) says they should go. But there's a catch: the table is a virtual world where physics rules apply. If you put the glass inside the plate, or if the knife is floating in mid-air, the simulation breaks, and the robot can't do its job.

The paper introduces STABLE, a new "brain" designed to solve this problem. It acts like a two-person team working together to build these perfect, physics-ready tables.

The Problem: The "Magic" vs. The "Physics"

Previous methods tried to use one super-smart AI (a Large Language Model, or LLM) to do everything. Think of this LLM as a brilliant storyteller who knows exactly what a "dinner table" looks like in a story. However, this storyteller is terrible at math and 3D geometry.

The Result: The storyteller might say, "Put the apple on the left of the banana," but in the 3D world, the apple ends up inside the banana (a collision) or hovering 2 inches above it (floating). The scene looks good in a story, but it's a disaster for a robot trying to pick it up.

The Solution: The STABLE Team

STABLE splits the job into two specialized roles, working in a loop:

1. The Semantic Reasoner (The "Architect")

Who they are: A fine-tuned AI that is great at understanding language and instructions.
What they do: They read the task (e.g., "Put the cup next to the saucer") and draw a rough sketch of the table. They decide what objects go there and roughly where.
The Analogy: Imagine an architect drawing a blueprint. They know the kitchen needs a sink and a stove, and they place them in the right rooms. But their blueprint is just lines on paper; they haven't checked if the stove is heavy enough to sit on the floor or if the sink is actually inside the wall.

2. The Physics Corrector (The "Inspector")

Who they are: A specialized AI trained on the laws of physics and 3D shapes.
What they do: They take the Architect's rough sketch and fix the physical errors. They nudge the objects so they don't overlap, ensure they are resting on the table (not floating), and make sure they are stacked stably.
The Analogy: This is like a construction inspector. They look at the blueprint and say, "Hey, that stove is floating! Let's drop it down until it hits the floor." Or, "That sink is inside the wall; let's pull it out so it doesn't crash." They use a special "mathematical ruler" (called Signed Distance Functions) to measure exactly how close objects are to each other to prevent collisions.

How They Work Together: The "Progressive" Dance

Instead of doing the whole table at once, STABLE builds it in layers, like stacking blocks:

Step 1: The Architect places the most important items (the ones the robot needs to interact with, like the cup).
Step 2: The Inspector immediately checks those items, fixing any floating or collisions.
Step 3: The Architect adds the background items (like a napkin or a fruit bowl) around the fixed items.
Step 4: The Inspector checks again, making sure the new items don't push the old ones into the air.

They keep alternating like this until the whole table is full. This ensures that by the time the table is done, it is both true to the instructions (the Architect's job) and physically solid (the Inspector's job).

Why This Matters

The paper shows that STABLE is much better than previous methods at two things:

No Collisions: It almost never puts objects inside each other.
No Floating: It almost never leaves objects hovering in mid-air.
Following Orders: It follows the specific instructions (like "left of the banana") much better than methods that try to "fix" the scene after the fact, which often messes up the original plan.

In short, STABLE is a system that combines a creative writer (who knows what the scene should look like) with a rigorous engineer (who knows how gravity and shapes work) to create digital tables that are ready for robots to use immediately.

Technical Summary: STABLE – Simulation-Ready Tabletop Layout Generation via a Semantics–Physics Dual System

1. Problem Statement

The generation of simulation-ready tabletop scenes from natural language task instructions is a critical component for training and evaluating Embodied AI. While Large Language Models (LLMs) have shown promise in interpreting high-level semantics to generate structured scene layouts, they suffer from inherent limitations in 3D spatial reasoning. Consequently, LLM-only approaches frequently produce scenes with physical impossibilities, such as object interpenetration (collisions), floating artifacts, or unstable stacking.

Existing attempts to address these issues fall into two categories, both with significant drawbacks:

LLM-centric methods: Often generate semantically aligned but physically invalid scenes.
Post-hoc optimization: Methods that apply physics-based solvers after generation can resolve collisions but often incur heavy computational costs, fail to find feasible solutions in highly cluttered scenes, and may disrupt the original semantic layout, causing the scene to drift from the user's task instructions (e.g., moving an object to the wrong side of a table to resolve a collision).

The core challenge is to generate tabletop layouts that strictly adhere to task instructions while ensuring physical plausibility (simulation readiness) without compromising scene diversity or semantic alignment.

2. Methodology: The STABLE Framework

The authors propose STABLE, a "Semantics–Physics" dual-system framework designed to decouple semantic layout generation from physics-aware pose correction. The system operates on a progressive inference paradigm, alternating between two complementary modules.

2.1. Semantic Reasoner (SR)

The Semantic Reasoner is a fine-tuned Large Language Model (based on Qwen3-8B) trained on a structured tabletop scene dataset (MesaTask-10K).

Function: It interprets task instructions to generate coarse, task-aligned scene layouts.
Progressive Generation: Unlike standard one-shot generation, STABLE employs a three-stage progressive construction process:
1. Task-oriented objects ( $O_t$ ): Objects explicitly specified in the instruction.
2. Important background objects ( $O_B$ ): Objects in physical contact or close proximity to task objects.
3. Secondary background objects ( $O_b$ ): Remaining contextual clutter.
Output: The SR outputs a structured JSON layout specifying object identities, sizes, and initial poses. It is trained to output these stages sequentially, strengthening instruction grounding and reducing the risk of missing critical objects.

2.2. Physics Corrector (PC)

The Physics Corrector is a lightweight, flow-based denoising model designed to refine the coarse poses generated by the SR.

Function: It predicts pose updates (3D translation and yaw rotation) to eliminate physical violations while preserving object identities, sizes, and semantic descriptions.
Geometry-Aware Conditioning: To handle complex relationships like stacking and containment, the PC is conditioned on explicit geometric features. It retrieves 3D assets, scales them to predicted sizes, samples surface point clouds, and encodes them using a frozen PointTransformerV3 encoder.
Flow Matching: The model uses Flow Matching to learn a continuous vector field that transports a noisy coarse pose to a physically valid target pose.
Physical Constraints (Loss Functions): To ensure simulation readiness, the training objective includes differentiable mesh-level constraints:
- Object-Object SDF Loss: Penalizes interpenetration between objects using Signed Distance Fields (SDF).
- Object-Table SDF Loss: Prevents objects from penetrating the tabletop.
- Support-Contact Loss: Encourages stable resting contact to eliminate floating artifacts, ensuring objects are supported by the table or other objects within a small tolerance.

2.3. Inference Pipeline

The system operates via an alternating loop:

The Semantic Reasoner expands the scene by adding a new stage of objects (e.g., from $O_t$ to $O_t \cup O_B$ ).
Assets are retrieved based on the new objects.
The Physics Corrector immediately refines the poses of the entire current scene to resolve any new collisions or floating issues introduced by the expansion.
The corrected layout serves as the context for the next stage of semantic expansion.

This pipeline allows for batched processing where different scenes can be at different stages of the loop, improving throughput.

3. Key Contributions

Dual-System Framework: STABLE is the first framework to explicitly decouple semantic layout generation (via LLM) from physics-aware pose correction (via a learned flow-based model), directly mapping task instructions to simulation-ready layouts.
Geometry-Aware Pose Correction: The introduction of a denoising target conditioned on mesh-level point clouds allows the system to resolve complex spatial relationships (stacking, containment) that bounding-box approximations cannot handle.
Progressive Inference: The staged generation strategy (Task $\to$ Important Background $\to$ Secondary Background) mitigates error accumulation and enables incremental scene editing.
Mesh-Level Physical Constraints: The integration of SDF-based collision and support losses during training ensures that the generated scenes are physically valid without relying on expensive post-hoc optimization.

4. Experimental Results

The authors evaluated STABLE on the MesaTask-10K dataset, comparing it against LLM-based baselines (MesaTask, Holodeck-Table, I-Design-Table), post-processing methods (Steerable), and proprietary models (GPT-4o).

Physical Validity: STABLE achieved a 0% Object Collision (OC) rate and 0% Floating rate, significantly outperforming baselines which suffered from high collision rates (e.g., MesaTask had 15.6% OC) or required post-processing that degraded task alignment.
Task Alignment: STABLE achieved the highest Align with Task (AwT) score (99.4%) and Align with Scene Graph (AwS) score (99.0%), demonstrating superior adherence to instructions compared to methods that drift during optimization.
Robustness: In collision-resolution tests, STABLE's Physics Corrector successfully resolved layouts with up to 40 initial collisions, whereas iterative post-hoc optimizers (MesaTask, Steerable) failed as collision counts increased.
Ablation Studies: Removing any physical constraint (SDF or support loss) significantly degraded simulation readiness. Similarly, removing the progressive construction strategy reduced task alignment, confirming the necessity of both components.
Downstream Applications: The method demonstrated effectiveness in scene rearrangement tasks and generalization to unseen tabletop types (e.g., TV stands, nightstands) and unseen object assets.

5. Significance and Claims

The paper claims that STABLE addresses a critical bottleneck in Embodied AI: the lack of high-quality, diverse, and physically valid synthetic data for robotic manipulation. By combining the semantic understanding of LLMs with the geometric rigor of learned physics correction, STABLE generates scenes that are not only semantically rich but also immediately usable in physical simulators.

The authors emphasize that their approach avoids the trade-off seen in prior work, where improving physical validity often came at the cost of semantic alignment. STABLE maintains strict task grounding while ensuring simulation readiness, offering a scalable solution for generating training data for generalist robotic policies. The work is presented as a step toward more robust and reliable synthetic data generation for embodied intelligence, though the authors acknowledge limitations regarding full 6-DoF pose prediction and the current focus on tabletop environments.