💻 computer science

TabletopGen: Tabletop Scene Generation and Interactive Simulation for Robotic Manipulation

TabletopGen is a training-free, automated engine that generates physically plausible and interactive tabletop scenes from text or single images to enable large-scale, high-fidelity robotic manipulation data synthesis and zero-shot real-world policy transfer.

Original authors: Ziqian Wang, Yonghao He, Licheng Yang, Wei Zou, Hongxuan Ma, Liu Liu, Wei Sui, Yuxin Guo, Hu Su

Published 2026-07-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ziqian Wang, Yonghao He, Licheng Yang, Wei Zou, Hongxuan Ma, Liu Liu, Wei Sui, Yuxin Guo, Hu Su

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot how to tidy up a messy kitchen counter. To do this safely and cheaply, you'd rather practice in a video game (a simulation) than risk breaking real dishes. But here's the problem: most video game worlds are either too simple (like a flat table with floating objects) or too messy (objects clipping through each other, defying gravity). If the robot practices on a broken world, it learns bad habits and fails when it tries to do the job in real life.

TabletopGen is a new tool that acts like a "magic architect" for these robot training worlds. It builds realistic, physics-compliant tabletop scenes from scratch, just by reading a text description or looking at a single photo.

Here is how it works, broken down into three simple steps:

1. The "Clay Sculptor" (Generative Instance Extraction)

Imagine you have a photo of a cluttered desk. You want to turn every item in that photo (a coffee mug, a laptop, a stapler) into a 3D object you can pick up.

The Problem: If you just copy the photo, the objects look flat or have holes in them because you can't see the back of them.
The TabletopGen Solution: It acts like a skilled sculptor. It looks at the photo, identifies every item, and "fills in the blanks" to create a complete, solid 3D model for each one. It then stands them all up straight, like a potter aligning clay pots on a wheel, so they are ready to be placed on a table.

2. The "Feng Shui Master" (Pose and Scale Alignment)

Now you have a pile of 3D objects. You need to arrange them on the table so they look like the photo and don't float in mid-air or sink through the wood.

The Problem: If you just guess where to put them, a book might be upside down, or a cup might be floating two feet above the table.
The TabletopGen Solution: It uses two special tools to get the arrangement perfect:
- The Rotation Tuner (DRO): It spins each object slightly until its shape and shadows match the original photo perfectly.
- The Top-View Planner (TSA): It imagines looking at the table from directly above (like a drone). It uses "common sense" physics (e.g., "a cup sits on a table, not inside it") to figure out exactly how big the objects should be and where they should sit so they don't crash into each other. It picks one reliable object as a "ruler" to ensure everything is the right size.

3. The "Game Builder" (Interactive Simulation)

Once the objects are arranged, the tool drops them into a physics engine (a video game engine).

The Result: It turns the scene into a playable level. The robot can now try to "pick up" the cup or "move" the book. If the robot tries to push a book off the table, it falls. If it tries to push a book through a wall, it bounces off.
The Bonus: Because the objects are separate and independent, you can easily swap a coffee mug for a teapot, or move a laptop to the other side, creating thousands of new practice scenarios instantly without rebuilding the whole world.

Why Does This Matter?

The paper proves that robots trained in these "magic architect" worlds can actually do the job in the real world.

The Test: They took a photo of a real table, built a simulation of it, trained a robot arm in the simulation to move fruit and blocks, and then told the robot to do the exact same task on the real table.
The Outcome: The robot succeeded! It didn't need any extra practice on the real table. This shows that the "magic architect" built a world that was accurate enough to teach the robot real-world skills.

In short: TabletopGen is a tool that turns a single photo or a sentence into a perfect, physics-accurate training ground for robots, allowing them to learn complex tasks quickly and safely without needing expensive real-world data collection.

Technical Summary: TabletopGen

Problem Statement

Robotic manipulation, particularly in tabletop environments, relies heavily on large-scale, high-quality interaction data for training Vision-Language-Action (VLA) models. While real-world data collection is costly and difficult to scale, simulation offers a promising alternative. However, existing 3D scene generation methods fail to produce simulation-ready tabletop environments effectively.

Retrieval-based methods suffer from insufficient diversity and geometric mismatches due to fixed asset libraries.
LLM-based text-driven methods typically focus on room-scale layouts, failing to handle the high-density, small-object functional arrangements and strong semantic constraints unique to tabletops.
Single-image reconstruction methods often produce physically implausible layouts (e.g., interpenetration, floating objects) and incomplete meshes due to single-view occlusions and a lack of physical priors.

Consequently, these scenes cannot be reliably imported into physics simulators for motion planning, creating a bottleneck for automated manipulation data synthesis.

Methodology

TabletopGen is a training-free, fully automated framework that generates high-fidelity, instance-level interactive tabletop scenes from a text description or a single image. The framework operates in three progressive stages:

1. Generative Instance Extraction

The system first converts text inputs into detailed prompts and synthesizes a reference image (or uses a provided image). It then extracts and reconstructs 3D assets:

Segmentation & Completion: A Multimodal Large Language Model (MLLM) performs open-vocabulary category recognition, combined with GroundedSAM-v2 for instance masking. To address occlusions, a multimodal generative model completes and redraws instance masks rather than using traditional inpainting.
3D Reconstruction & Canonicalization: Completed 2D instances are reconstructed into 3D meshes using Image-to-3D models. Crucially, the system applies rotation correction to align each object's local vertical axis with the world Z-up direction, ensuring the assets are "Z-up canonicalized" and directly placeable.

2. Pose and Scale Alignment

This is the core innovation, decoupling the estimation of rotation, translation, and scale into two specialized phases to ensure physical plausibility:

Differentiable Rotation Optimizer (DRO): This phase optimizes the rotation ( $r_i$ ) of each instance around the vertical axis. It uses a differentiable renderer to generate a textured image, soft silhouette, and edge map, minimizing a tri-modal loss (shape consistency via soft IoU, contour matching via Chamfer loss, and appearance similarity via DINOv2 features) against the target instance.
Top-View Spatial Alignment (TSA): This phase infers translation ( $t_i$ $t_{i}$ ) and scale ( $s_i$ $s_{i}$ ) using physical priors:
- Stacking Priors: An MLLM infers stacking relations to prevent floating objects or vertical interpenetration.
- Top-View Synthesis: A generative model synthesizes a top-view of the scene to extract 2D bounding boxes, ensuring objects stay within tabletop boundaries.
- RMA-Score: A "Ratio-Matched and Area-Weighted Score" selects a reliable anchor object based on aspect ratio consistency and pixel area. This anchor determines a global scaling factor to infer metric 3D scales and horizontal translations for all objects.

3. Interactive Simulation

The aligned instances are assembled in a physics simulator (Isaac Sim). Collision geometries are generated via convex decomposition, and physical properties (gravity, friction) are enabled. The framework supports:

Task Generation: Automatic generation of manipulation tasks (e.g., "put the cup in the box") based on semantic labels.
Data Synthesis: Execution of collision-free trajectories to collect multimodal data (RGB, joint states, actions).
Scene Expansion: In-scene editing (add, remove, replace, rearrange) to generate diverse scene variants without re-running the full pipeline.

Key Contributions

TabletopGen Framework: An automated, training-free engine that generates simulation-ready tabletop scenes from text or single images, achieving visual realism, instance-level interactivity, and physical plausibility simultaneously.
Novel Alignment Method: A decoupled pose and scale alignment approach using DRO for precise rotation optimization and TSA for physically constrained translation and scale inference, overcoming the limitations of single-view reconstruction.
Comprehensive Validation: Extensive experiments demonstrating state-of-the-art performance in visual fidelity and physical validity, validated by a zero-shot real-to-sim-to-real policy transfer.

Results

Quantitative Performance: In comparisons with image-driven baselines (ACDC, Gen3DSR, MIDI) and text-driven methods (MesaTask, Holodeck-table), TabletopGen achieved the highest scores in visual/perceptual metrics (LPIPS, DINOv2, CLIP) and GPT-4o evaluations.
Physical Plausibility: TabletopGen significantly reduced collision rates. While baselines exhibited scene-level collision rates (Col_S) between 67% and 98%, TabletopGen achieved a Col_S of only 7.69% and a mesh-level collision rate (Col_O) of 0.42%.
User Study: In a study with 128 participants, TabletopGen was preferred in 83.13% of cases, significantly outperforming baselines in Visual Fidelity, Image Alignment, and Physical Plausibility.
Real-World Transfer: In a zero-shot real-to-sim-to-real experiment, policies trained solely on synthetic data generated by TabletopGen were deployed on a real AgileX PiPER arm. The policies achieved high success rates (e.g., 84% for placing a mango, 80% for placing a strawberry) without any real-world fine-tuning, validating the physical accuracy of the generated scenes.

Significance

The paper claims that TabletopGen serves as a reliable data engine for robotic manipulation. By solving the bottleneck of generating physically plausible, instance-level interactive scenes, it enables the low-cost, scalable synthesis of multimodal manipulation data. This capability is critical for training generalizable VLA models and supports the "sim-to-real" paradigm, allowing robots to learn complex manipulation policies in simulation that transfer effectively to the physical world without extensive real-world data collection. The framework's ability to support scene expansion through in-scene editing further offers a scalable route to covering diverse simulation environments for embodied AI.