🤖 AI

WorldCoder-Bench: Benchmarking Physically Grounded 3D World Synthesis

This paper introduces WorldCoder-Bench, a comprehensive benchmark featuring 2,026 expert-curated tasks and the StateProbe execution-based protocol to evaluate and verify the ability of large language models to synthesize physically grounded, interactive 3D worlds in browser-native environments, revealing that current frontier models struggle significantly with maintaining state consistency and interaction chains.

Original authors: Shuo Lu, Yinuo Xu, Kecheng Yu, Siru Jiang, Yongcan Yu, Yubin Wang, Haitao Yang, Yuxiang Zhang, Bin Wang, Ran He, Jian Liang

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shuo Lu, Yinuo Xu, Kecheng Yu, Siru Jiang, Yongcan Yu, Yubin Wang, Haitao Yang, Yuxiang Zhang, Bin Wang, Ran He, Jian Liang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a robot architect to build a virtual playground based on a simple description you give it, like "Make a basketball game where the ball bounces realistically."

In the past, if you asked an AI to build a website, you could just look at the screen. If the buttons looked right and the colors were nice, you assumed it worked. But building a 3D world (like a video game or physics simulator) inside a web browser is different. It's like building a house where the walls are made of invisible glass. You can see the outside, but you can't tell if the foundation is solid, if the doors actually open, or if gravity is working correctly just by looking at a picture.

This paper introduces WorldCoder-Bench, a new way to test if AI can actually build these working 3D worlds, not just pretty pictures of them.

Here is the breakdown of their approach using simple analogies:

1. The Problem: The "Black Box" Canvas

When an AI builds a 3D world, it writes code that runs inside a hidden area of the browser called a <canvas>.

The Old Way: Previous tests were like hiring a critic who only looks at a photograph of the playground. They might say, "The slide looks blue, so it's good!" But they can't see if the slide is actually attached to the ground or if the ball rolls off the edge.
The Reality: The AI might draw a perfect-looking basketball hoop, but if the code is wrong, the ball might float through the hoop without scoring, or the hoop might disappear when you touch it. The old tests missed these "invisible" failures.

2. The Solution: The "State Probe" (The Invisible Inspector)

The authors created a new tool called StateProbe. Instead of just taking a photo, this tool is like an invisible inspector who walks inside the virtual world.

How it works: The inspector has a secret checklist (a "contract") written by human experts. It doesn't just look at the scene; it reaches inside the code to check the "vital signs" of the world.
What it checks:
- Physics: Is the ball actually falling at the right speed?
- Interaction: When I click the button, does the game actually register the click, or is the button just a drawing?
- State: If I score a point, does the score counter actually go up, or is it stuck at zero?

To make sure the inspector is tough, they used a trick called Mutation Testing. They deliberately broke the AI's code in small ways (like changing gravity to be super weak or hiding the score button) to see if the inspector would catch the mistake. If the inspector missed the break, they fixed the inspector. This ensures the test is strict and fair.

3. The Test: WorldCoder-Bench

They built a massive test suite with 2,026 different challenges.

The Tasks: These range from simple things (making a ball bounce) to complex things (simulating chemical reactions or building a game where you have to aim a basketball).
The Rules: The AI gets a natural language instruction (e.g., "Build a game...") and must produce a single web page. It doesn't get to see the secret checklist or the inspector's rules.

4. The Results: The AI is Still Learning

They tested the top 9 AI models in the world. The results were surprising:

The Score: Even the best AI only passed about 28% of the tests.
The Illusion: Many AIs produced worlds that looked perfect in a screenshot but failed the "invisible inspector" test. They built the scenery but forgot the rules of physics or how to connect the buttons to the game logic.
The Main Mistakes: The AI usually got the "look" right but failed at:
- Schema Drift: The AI changed the rules of the game without telling the inspector (e.g., the ball is supposed to be red, but the AI made it blue and didn't update the score).
- Broken Chains: The AI built a door, but it didn't connect the handle to the door, so it wouldn't open.

5. The "Value" Check: Is it Worth the Money?

The authors also calculated if using these AIs saves money compared to hiring a human developer.

The Twist: Even though the AIs fail often, they are so cheap and fast that for simple tasks, they still save a lot of money. It's like having a very cheap, very fast apprentice who gets the easy jobs done quickly but needs a human to fix the complex ones.
The Metric: They created a "Return on Automation" score. For easy tasks (like making a cool visual effect), the AI is a great deal. For hard tasks (like complex physics), the AI is still too unreliable to replace a human.

Summary

WorldCoder-Bench is a reality check for AI. It stops us from being fooled by pretty pictures and forces the AI to prove that its 3D worlds actually work under the hood. Right now, the best AI models are like talented artists who can paint a perfect picture of a car but haven't learned how to build an engine that actually runs. We are getting closer, but there is still a long way to go before they can build fully functional, interactive worlds on their own.

Technical Summary: WorldCoder-Bench

Problem Statement

Current Large Language Models (LLMs) are increasingly tasked with generating executable web applications from natural language. However, existing benchmarks for web generation primarily evaluate static interfaces or 2D visual outputs (pixels, DOM nodes). This approach is insufficient for browser-native 3D worlds (e.g., built with Three.js), where the core logic—physics simulations, collision detection, and state synchronization—occurs within an opaque <canvas> and JavaScript runtime.

The paper identifies a critical "blind spot": a generated 3D world may look visually plausible (passing screenshot or DOM-based checks) while failing fundamental behavioral requirements, such as objects colliding correctly, energy dissipating at expected rates, or UI widgets staying synchronized with the engine state. Existing evaluation paradigms (screenshots, DOM walkers, or external agent exploration) cannot access the internal runtime state required to verify these mechanics, leading to high false-positive rates in assessing code correctness.

Methodology

1. WORLDCODER-BENCH: The Benchmark

The authors introduce WORLDCODER-BENCH, a benchmark comprising 2,026 expert-curated tasks designed to evaluate autonomous, physically grounded 3D world synthesis.

Task Structure: Each task requires the model to generate a single, self-contained HTML file using Three.js based on a natural-language prompt. Tasks may include optional .glb assets.
Taxonomy: Tasks are categorized into three macro-intents:
- Simulation: Physically evolving systems (e.g., soft bodies, chemistry, physics).
- Rendering: Controllable visual effects (e.g., shaders, materials, post-processing).
- Application: Goal-oriented interactive use (e.g., games, architecture, animation).
Splits:
- WORLDCODER-CORE: 205 high-difficulty tasks for the primary leaderboard.
- WORLDCODER-EXTENDED: 1,621 tasks for large-scale comparison.
- WORLDCODER-ROBUST: 615 variants of CORE tasks with randomized physical constants (gravity, elasticity), object counts, and asset filenames to test generalization over template memorization.
- WORLDCODER-DEV: 200 public tasks with released contracts for debugging.

2. STATEPROBE: Execution-Based Verification

To evaluate the "hidden" behavior of 3D worlds, the authors propose STATEPROBE, an execution-based protocol that moves beyond visual observation.

Runtime Interface: Generated programs must expose a standardized state interface (e.g., window.__3D_STATE__) reporting variables like coordinates, velocities, and interaction flags.
Hidden Contracts: Each task is paired with a hidden, executable behavioral contract authored by domain experts. These contracts define expected affordances, state transitions, and numerical invariants (e.g., conservation laws).
Mutation Hardening: Contracts are rigorously calibrated using mutation testing. Common defects (stale state updates, broken asset loading, swapped event targets) are injected into reference implementations. A contract is only admitted if it successfully rejects these mutated variants, ensuring that a "pass" reflects genuine correctness rather than loose thresholds.
Execution Flow: Programs run in a headless Chromium browser. A scripted action sequence drives the world, and STATEPROBE snapshots state before and after each step to verify deltas against the hidden contract.

3. Metrics

Beyond binary pass/fail, the paper introduces:

Coverage Metrics:
- Affordance Coverage (A-Cov): Presence of required objects/controls.
- State Coverage (S-Cov): Reachability of intermediate states.
- Transition Coverage (T-Cov): Correctness of action-triggered state changes.
- Verification Coverage (V-Cov): The primary metric; the proportion of hidden assertions passed.
Utility Metrics:
- Return on Automation (RoA): Quality-adjusted human labor value saved per dollar of model cost.
- Time Efficiency Multiplier (TEM): Human-hours saved per hour of model generation time.

Key Contributions

WORLDCODER-BENCH: The first benchmark to score models on the behavioral correctness of generated Three.js worlds rather than visual plausibility, covering 2,026 tasks across 15 fine-grained domains.
STATEPROBE: A novel evaluation protocol using hidden, mutation-hardened contracts and runtime state probing to verify physics, asset integration, and state synchronization.
Utility Metrics: The introduction of RoA and TEM, which combine correctness, inference cost, and latency to measure practical economic value, penalizing cheap but incorrect outputs.
Empirical Analysis: A comprehensive benchmark of nine frontier models (including GPT-5.4, Claude Opus 4.6, and Qwen3.6-Plus) revealing significant gaps between visual appearance and functional correctness.

Experimental Results

The authors evaluated nine frontier models on WORLDCODER-CORE and WORLDCODER-ROBUST:

Low Verification Coverage: The best-performing model (GPT-5.4) achieved only 27.8% V-Cov on CORE and 19.9% on ROBUST. No system exceeded 30% on the hidden contracts.
Failure Modes: Failures were dominated by State Schema Drift (42.8%) and Broken Interaction Chains (40.8%), rather than missing scene elements or physics calculation errors. Models often generated visually plausible scenes but failed to synchronize runtime state with user actions.
Evaluation Paradigm Failure: External evaluation methods (DOM-only, Screenshot+VLM, Agent exploration) showed near-zero or negative correlation with ground-truth V-Cov. For instance, DOM-based scoring had a Kendall $\tau_b$ of -0.021, and agent-based exploration incorrectly passed 45.6% of severely defective outputs.
Robustness: The gap between proprietary and open-weight models widened on the ROBUST split (6 points vs. 2.5 points on CORE), suggesting open models rely more heavily on template recall than mechanism understanding.
Economic Value: Despite low V-Cov, "cheap" models (e.g., DeepSeek-V4-Flash) showed high RoA ($19,150 per dollar) due to low inference costs, while faster models (GPT-5.4) led in TEM (68.9x). Rendering tasks were significantly easier than Simulation or Application tasks.

Significance and Claims

The paper claims that WORLDCODER-BENCH and STATEPROBE are necessary to diagnose the specific failure modes of LLMs in 3D coding, which are currently masked by visual-only evaluators.

Diagnostic Value: The benchmark reveals that the primary barrier to reliable 3D world synthesis is not the generation of static geometry, but the maintenance of action-synchronous runtime state contracts and physical invariants.
Limitations of Current Models: Even the strongest models fail to consistently produce executable worlds that obey hidden behavioral contracts, indicating that "looks right" is no longer a sufficient proxy for "works right."
Future Direction: The authors position the benchmark as a diagnostic platform to guide the development of agents capable of constructing executable worlds, not just plausible scenes. They emphasize that progress must be measured against behavioral reliability, particularly in the "long tail" of complex systems and physics-heavy tasks where current models struggle.