💻 computer science

3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code

This paper introduces 3DCodeBench, a comprehensive benchmark and ranking platform designed to evaluate the capabilities of vision-language model agents in generating procedural 3D models from text and image prompts, revealing that while test-time scaling improves performance, current models struggle with API mismatches and geometric coherence, highlighting the need for better coding data and execution environments.

Original authors: Yipeng Gao, Lei Shu, Genzhi Ye, Xi Xiong, Ameesh Makadia, Meiqi Guo, Laurent Itti, Jindong Chen

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yipeng Gao, Lei Shu, Genzhi Ye, Xi Xiong, Ameesh Makadia, Meiqi Guo, Laurent Itti, Jindong Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to build a complex piece of furniture, like a crab-shaped chair or a floating tree. In the past, you'd need to hire a master carpenter who knows exactly how to cut every piece of wood and assemble it perfectly. Today, we are trying to teach AI to be that carpenter, but instead of using a saw and hammer, the AI writes computer code to build the object inside a 3D software program called Blender.

This paper, titled 3DCodeBench, is essentially a giant "driver's license test" for these AI carpenters. Here is the breakdown of what they did and what they found, using simple analogies.

1. The Problem: AI Can Talk, But Can It Build?

We have AI models that are great at writing stories or answering questions. We also have AI that can generate static 3D pictures. But there is a gap: Procedural 3D modeling. This means the AI writes a script (a set of instructions) that tells the computer, "Start with a cube, cut a hole here, add a fin there, and make it look like a crab."

The problem is that writing this code is hard. If the AI makes a tiny mistake in the code, the computer crashes, or the result is a floating, disconnected mess of shapes that doesn't look like a real object.

2. The Solution: A New Test Track (3DCodeBench)

The researchers built a massive testing ground called 3DCodeBench. Think of it as a driving school with 212 different "obstacle courses" (like building a fish, a sink, or a maple leaf).

The Dataset: They didn't just make up random shapes. They took a huge library of existing, perfect 3D models and reverse-engineered them to create the "correct" code. This gave them 26,000 pairs of "Instructions + Correct Code + Final Object."
The Test: They asked 12 different top-tier AI models to look at a prompt (like "Make a crab") and write the code to build it.
The Arena (3DCodeArena): Since computers can't always tell if a 3D crab looks "cool" or "weird," they built a public voting arena. Humans look at two AI-generated crabs side-by-side and vote on which one looks better. This creates a leaderboard (Elo ranking) just like in chess or video games.

3. What They Found: The AI is Getting Better, But Still Clumsy

Finding 1: The "Syntax" vs. The "Structure"
The AI is surprisingly good at writing code that runs without crashing. It's like a student who can write a perfect recipe on paper. However, the resulting dish (the 3D object) often falls apart.

The Issue: The AI often creates parts that are floating in mid-air or disconnected from the main body. It understands the words of the code but struggles with the physics of how 3D objects actually fit together.

Finding 2: "Thinking Harder" Helps (But Only Sometimes)
The researchers tested what happens if they tell the AI to "think longer" before answering (giving it more time to plan).

The Result: For the smaller, lighter AI models, giving them more "thinking time" was like giving a student a calculator; it helped them fix simple mistakes and write working code.
The Limit: For the biggest, smartest models, they were already so good at the basics that extra thinking time didn't help much. They just needed to be told to "try again" if they failed.

Finding 3: The "Do-Over" Button is Magic
This is the most important discovery. When an AI fails to build the object, the researchers let it see the error message (like a "syntax error" or "crash") and try again.

The Result: This simple "try again" loop turned a 50% success rate into a 97% success rate. It's like a student who fails a math test, looks at the answer key to see where they went wrong, and then immediately passes the re-test. The AI doesn't need to be smarter; it just needs a chance to fix its specific mistakes.

Finding 4: Humans vs. Machines
They asked: "Can an AI judge how good another AI's 3D model is?"

The Result: If the AI judge looks at the final 3D picture, it can agree with human voters about 75% of the time.
The Catch: If the AI judge only looks at the code (without seeing the result), it gets much worse. This proves that to judge 3D modeling, you really need to see the final product, not just read the instructions.

4. The Bottom Line

The paper concludes that while AI is getting very good at writing the code to build 3D objects, it still struggles with the logic of making those objects look physically real and connected.

However, the biggest breakthrough isn't a new AI model; it's the process. By letting the AI run its code, see where it breaks, and fix it (a "multi-turn" process), we can get near-perfect results even with current models.

In short: We have built a gym (3DCodeBench) to train AI carpenters. We found that they are great at following instructions but need a coach to point out their mistakes so they can fix the floating, broken pieces. Once they get that second chance, they can build almost anything.

Technical Summary: 3DCodeBench

Problem Statement

Procedural 3D modeling via code is a critical paradigm for generating deterministic, engine-ready, and precisely editable assets for gaming, industrial design, and robotics. However, authoring such content requires deep expertise in 3D software APIs, parametric design, and geometric reasoning, creating a significant barrier to entry. While Vision-Language Models (VLMs) and coding agents show potential for automating this process, the field lacks a standardized, reliable benchmark to evaluate their capabilities. Existing methodologies suffer from four core limitations:

Data Scarcity: A lack of aligned procedural data (text/image prompts paired with executable code and ground-truth meshes).
Simplistic Environments: Current benchmarks often focus on editing pre-built scenes or manipulating low-complexity voxel grids rather than generating complex geometry from scratch.
Lack of Iteration: Existing frameworks rely on single-shot evaluations, ignoring the iterative refinement loops essential to real-world 3D design workflows.
Absence of Standardized Metrics: The field lacks comprehensive perception metrics and a systematic platform for human preference voting to evaluate the perceptual quality of 3D shapes.

Methodology

The authors introduce 3DCodeBench, a comprehensive benchmark and evaluation framework designed to address these limitations.

1. Agentic Data Curation Pipeline

To construct a high-quality dataset, the authors developed an automated pipeline that transforms deeply nested procedural factories from the Infinigen simulator into standalone Blender 5.0 Python scripts. This pipeline employs:

Skills Library: Agents use specialized tools including a Code Simplifier (to reduce nested code), a Simulator (to execute code in a sandboxed Blender 5.0 environment), a Visual Critic (a VLM comparing multi-view renders against references), and a Mesh Analyzer (checking for structural validity).
Experience Library: A knowledge base containing class deduplication, parts assembly templates, Blender 5.0 API migration rules, and code organization standards to prevent recurring errors.
Human-in-the-Loop Verification: Human annotators verify execution reliability, semantic accuracy of captions, and visual alignment. Only triplets passing this audit are included.

The resulting dataset comprises 26,000 text/image prompt ↔ standalone code ↔ 3D object triplets across 212 distinct categories (ranging from organic entities like crabs and flora to manufactured objects like furniture). The median script length is 387 lines, with some exceeding 1,000 lines, ensuring the benchmark tests complex geometric reasoning rather than simple primitive composition.

2. Evaluation Protocol

The benchmark evaluates VLMs on two axes:

Quantitative Metrics:
- Executability: Binary indicator of whether the script runs successfully in Blender 5.0 and produces a valid mesh.
- Perceptual Fidelity: Cosine similarity between rendered views and references using SigLIP-2 and DINOv3.
- Geometric Alignment: Chamfer Distance and Uni3D (3D-3D and cross-modal) similarities on exported GLB files.
- Metrics are reported in both conditional (averaged over successful runs) and penalized (assigning zero/penalty for failures) modes.
3DCodeArena: A public platform for pairwise human preference voting. It generates Elo rankings based on human votes comparing generated meshes, addressing the limitation of automated metrics in capturing perceptual quality.

3. Experimental Setup

The authors evaluated 12 frontier VLMs (including GPT-5.5, Gemini 3.1 Pro, and Claude Opus 4.7) across text-to-3D and image-to-3D tracks. Experiments included:

Single-shot vs. Multi-turn: Testing models with and without error-feedback loops (up to two retries with traceback information).
Agentic Harnesses: Wrapping models in native coding agents (e.g., Claude Code, Codex CLI) to allow autonomous debugging and iteration.
Ablation Studies: Analyzing the impact of "thinking budgets" (reasoning effort) and the number of input reference views.

Key Results

1. Correlation Between Metrics and Human Preference

The study found that SigLIP-2 view similarity is the strongest automated predictor of human preference (Pearson $r = 0.964$ ), followed closely by DINOv3 (Spearman $\rho = 0.972$ ). Executability alone is a weak predictor of human preference, indicating that while models can often generate code that runs, the resulting geometry frequently lacks physical plausibility or structural integrity.

2. The Impact of Test-Time Scaling

Thinking Budgets: Increasing the thinking budget significantly improves performance for lightweight reasoners (e.g., Gemini 3.1 Flash Lite gains ~19 executability points), while frontier models (e.g., Claude Opus 4.7) show diminishing returns, often plateauing at minimal effort levels.
Multi-turn Error-Feedback: This is the most effective driver for improvement. Multi-turn retries lifted aggregate executability from 69.2% to 97.2% across all models. The primary failure mode addressed was Blender 5.0 API mismatches, which models could self-correct once provided with error tracebacks.
Agentic Harnesses: While native coding agents further improved executability (reaching near 100% for top models), they did not significantly improve the conditional shape quality (geometry fidelity) compared to single-shot prompts. The harnesses primarily solved API syntax errors rather than enhancing geometric reasoning.

3. Qualitative Findings

Despite high executability rates with multi-turn refinement, models still struggle with physical plausibility. Common failure modes include disconnected geometric components, floating primitives, and incorrect structural alignments. The paper notes that models frequently produce "disconnected parts" even when the code executes successfully.

4. Cost-Quality Trade-offs

Analysis of the Pareto frontier reveals that Gemini models (specifically 3.1 Pro, 3.5 Flash, and 3 Flash) offer the best balance of cost and performance. GPT-5.5 achieves the highest Elo scores but at a significantly higher cost. Claude models generally fall below the frontier, scoring lower Elo at similar or higher costs.

Significance and Contributions

The paper claims three primary contributions:

3DCodeBench: A standardized, VLM-driven benchmark spanning 212 object classes and 26K code-mesh pairs, curated via an agentic pipeline with human verification. It moves beyond simple primitives to test complex, real-world geometric reasoning.
3DCodeArena: A systematic human-preference platform that establishes reliable Elo rankings, providing a necessary complement to automated metrics for evaluating perceptual quality.
Insights into Agentic Workflows: The study identifies that while VLMs can generate executable code, they lack deep physical-world understanding. Crucially, it demonstrates that test-time scaling (multi-turn refinement and error feedback) is the primary driver for successful procedural 3D generation, effectively mitigating API mismatches but not fully solving geometric reasoning gaps.

The authors conclude that advancing commercial VLMs for procedural 3D modeling requires high-quality procedural coding data and robust execution environments that provide high-fidelity feedback for iterative refinement. The release of 3DCodeBench and 3DCodeArena provides a foundational toolkit for this exploration.