💻 computer science

Coding Agent Is Good As World Simulator

This paper introduces an agentic framework that constructs physically plausible world models by iteratively generating and refining executable simulation code through coordinated planning, coding, visual review, and physics analysis, demonstrating superior performance over video-based approaches in physical accuracy, instruction fidelity, and visual quality.

Original authors: Hongyu Wang, Jingquan Wang, Bocheng Zou, Radu Serban, Dan Negrut

Published 2026-05-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hongyu Wang, Jingquan Wang, Bocheng Zou, Radu Serban, Dan Negrut

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to build a realistic movie scene where a car drives over a floating bridge on water, or a robot navigates a messy office.

The Old Way (Video Models):
Think of current AI video generators as incredibly talented painters. If you ask them to paint a car driving over water, they look at millions of photos and videos of cars and water. They guess what the next frame should look like based on patterns.

The Problem: They don't actually know physics. They are just guessing the colors and shapes. If the car hits a bump, the painter might accidentally make the car float through the bump, or the bridge might twist into a weird shape because the AI "forgot" how gravity works. It looks cool for a second, but the laws of physics break down quickly.

The New Way (This Paper's Approach):
The authors propose a different method: instead of painting the movie frame-by-frame, they ask the AI to write the instruction manual (code) for a physics engine.

Think of it like this:

The Old Way is like a magician trying to guess the next trick.
This Paper's Way is like hiring an engineer to build a real, working model in a lab.

How the "Coding Agent" Works

The paper describes a team of AI "agents" (specialized assistants) working together to build this simulation. They act like a construction crew:

The Planner (The Architect): You tell the team, "I want a robot in an office." The Planner breaks this down into a blueprint: "We need a floor, two tables, seven chairs, and a robot. The robot needs to be able to walk between them."
The Coder (The Builder): This agent takes the blueprint and writes actual computer code (using a tool called Project Chrono) that tells the computer exactly how to build the room, where to put the furniture, and how heavy everything is.
The Visual Reviewer (The Inspector): The code runs, and a video is generated. The Reviewer watches the video and says, "Wait, the robot is floating through the table. That's wrong."
The Physics Analyst (The Engineer): This agent checks the math. It says, "The table is too light, or the robot is too heavy. The code needs to be fixed."
The Loop: The Coder fixes the code, and they run it again. They keep doing this until the simulation works perfectly and obeys the laws of physics.

Why This Matters

The paper claims that by building the world through executable code rather than just predicting video frames, the simulation becomes:

Physically Accurate: Objects collide, fall, and bounce exactly as they should because the code enforces real physics rules.
Repairable: If something goes wrong, you can look at the code, find the mistake, and fix it. You can't easily "fix" a video if the physics are wrong; you just have to try to paint it again.
Reliable for Complex Tasks: The team tested this with a robot in an office, a military vehicle driving on rough terrain, and a car driving over a floating bridge on water (a complex mix of solid and liquid physics).

The Results

When they compared their "Coding Agent" method against advanced video-generation models (using a benchmark called WorldModelBench), their method scored higher on:

Instruction Fidelity: Did it do exactly what you asked? (Yes, because the code explicitly builds what you asked for).
Physical Laws: Did the car fall through the floor? (No, because the code prevents it).

The Catch

The paper admits this system isn't perfect yet. It can be slow and expensive (in terms of computer power) because it has to write code, run it, check it, and fix it repeatedly. Also, if the AI doesn't have a specific 3D model of an object in its library, it has to approximate it with a simple shape.

In short: Instead of asking AI to guess what a realistic world looks like, this paper teaches AI to build a realistic world using code, ensuring the physics are real, not just an illusion.

Technical Summary: Coding Agent Is Good As World Simulator

Problem Statement

World models have emerged as a paradigm for building interactive simulation environments, with recent video-based approaches demonstrating progress in generating visually plausible dynamics. However, these models typically infer dynamics from video and represent them in latent states rather than explicitly enforcing physical constraints. Consequently, their generated video rollouts often lack physical plausibility, exhibiting unstable contacts, distorted shapes, or inconsistent motion. A critical gap remains in constructing worlds where mechanics can be inspected, executed, and repaired, particularly for long-horizon interactions where preserving the physical state is as important as rendering the next frame. Existing physics simulators (e.g., MuJoCo, Project Chrono) offer explicit state and diagnostics but require users to manually choose assets, instantiate bodies, write code, and tune parameters, creating a bottleneck in world construction.

Methodology

The authors propose a multi-agent framework that constructs physics-based world models through executable simulation code rather than direct frame prediction. The system treats generated code as the world representation, specifying geometry, bodies, joints, contacts, materials, sensors, and numerical settings within a physics engine (Project Chrono). The framework operates via a closed-loop agent workflow:

Asset Library and Collision Representation: The system utilizes a unified asset library combining external 3D assets (for semantic/visual diversity) and simulator-native assets (for physical components). To manage computational costs, visual geometry is separated from collision geometry; high-resolution meshes are used for rendering, while physical interactions rely on decomposed convex hulls generated via the Approximate Convex Decomposition (CoACD) algorithm.
Multi-Agent Workflow:
- Plan Agent: Converts underspecified natural language prompts (and optional reference images) into a structured simulation plan. This agent extracts physical entities, infers scene topology using a defined predicate algebra (e.g., PLACE-ON, FLOATS-AT-SURFACE, FACING-TO), and resolves spatial relations into concrete simulator poses. It exposes this plan for user confirmation to ensure concrete choices (dimensions, time steps, camera placement) are validated before coding.
- Code Agent: Translates the approved plan into executable PyChrono code. It conditions generation on a curated skill library (implementation patterns for rigid bodies, joints, vehicles, etc.), a tool interface for querying resources, and a version-specific API index to prevent API drift.
- Execution and Review Loop: The generated code runs in the Chrono engine.
  - Visual Review Agent: Analyzes simulation videos to describe scene layout, dynamics, and visual inconsistencies.
  - Simulation Judge: Combines diagnostic logs, physical trajectory data, and visual evidence to determine validity. If the simulation fails (e.g., physics errors, visual mismatches), the judge returns a structured error report.
  - Iterative Repair: The Code Agent patches the current program based on feedback rather than regenerating from scratch. This loop continues until the simulation satisfies the plan and physical constraints.

Key Contributions

Multi-Agent Framework for World Simulation: The authors propose a system where agents collaboratively construct executable physics worlds through simulator-aware planning, skill-grounded code generation, execution feedback, visual review, and iterative fixing.
Executable World Representation: The framework integrates physics simulation directly into the world-construction process. Objects, joints, contacts, and numerical settings are represented as executable simulator programs, enabling explicit physical states, inspectable dynamics, and grounded interaction beyond frame-level visual prediction.
Demonstrated Effectiveness and Generality: The framework is evaluated across diverse tasks, including robot interaction in indoor environments, outdoor vehicle dynamics, and high-fidelity fluid–solid interaction (FSI).

Experimental Results

The framework was evaluated using three scenarios: a Go2 robot patrolling an office, an HMMWV driving on outdoor terrain, and a Polaris vehicle crossing a floating block on water (FSI).

Planning Robustness: An ablation study on the Plan Agent showed high success rates (100% Pass@1) for generating structured plans, both with and without reference images.
Resource Usage: End-to-end runs for successful simulations took between 24–30 minutes, with token usage ranging from ~1.68 million to ~6.34 million tokens, depending on scenario complexity.
Benchmark Evaluation (WorldModelBench): The framework was compared against a state-of-the-art video generation baseline (Wan2.2-TI2V-5B) on the WorldModelBench, which assesses instruction adherence, physical laws, and common sense.
- The multi-agent framework achieved higher total scores in all three scenarios.
- The improvement was statistically significant ( $p=0.0012$ ) in the FSI vehicle scenario.
- At the metric level, the framework showed a significant gain in instruction adherence ( $p=0.000059$ ), suggesting superior preservation of requested entities and scene constraints.
- Physical-laws and common-sense scores were comparable to the baseline, though the framework maintained a slight numerical edge.

Significance and Limitations

The paper posits that a coding agent can effectively act as a world simulator by constructing executable programs that define the physical world itself. This approach shifts the problem from frame prediction to simulator-aware world construction, allowing failures to be traced through execution logs and repaired via code modification rather than retraining or regenerating latent states.

The authors acknowledge several limitations:

Repair Monotonicity: The repair process is not guaranteed to improve monotonically and may cycle through different failure modes.
Asset Constraints: The finite asset library requires missing objects to be approximated by geometric proxies.
Code Constraints: Generation is limited by the skill library and API index, making unsupported sensors or custom solvers difficult to instantiate.
Evaluation Scope: The current evaluation covers a small number of scenarios and relies partly on human judgment for plan acceptance.

Future work directions identified include integrating 3D asset generation pipelines to handle long-tail elements, optimizing token usage through better state representations and caching, and exploring parallel execution to overcome current hardware bottlenecks.