Technical Summary: BuildArena

Problem Statement

Engineering construction automation aims to translate natural language specifications into physically viable structures, a task requiring complex integrated reasoning under strict geometric, physical, and structural constraints. While Large Language Models (LLMs) have demonstrated proficiency in mathematical reasoning, code generation, and following instructions, their capabilities for language-driven engineering construction remain largely unevaluated.

Existing benchmarks fall short in addressing this specific domain:

• Planning Benchmarks (e.g., PlanBench, PlanQA) operate in abstract, static environments lacking physical constraints.
• Physical Reasoning Benchmarks (e.g., PHYRE, PIQA) focus on understanding physics but neglect multi-step construction processes and assembly.
• 3D/CAD Generation Research often validates generation performance but rarely tests whether designs yield executable assemblies under realistic physical conditions.

There is a critical interdisciplinary gap: the absence of a framework to evaluate whether LLMs can effectively translate natural language into physically viable, assembled structures.

Methodology

The authors propose BuildArena, the first physics-aligned interactive benchmark designed to assess LLMs in engineering construction tasks. The framework consists of three core components:

1. Task Design

BuildArena defines three representative engineering task categories, each with three difficulty levels (Easy, Medium, Hard) calibrated against six engineering dimensions: Quantification, Robustness, Magnitude, Compositionality, Precision, and Ambiguity.

• Transport: Constructing machines for directional movement on a planar surface. Difficulty increases by removing explicit vehicle instructions and introducing cargo loads of varying sizes.
• Support: Designing bridges to span gaps of increasing width. Difficulty scales by increasing span and allowing modular construction (up to three substructures) without detailed assembly instructions.
• Lift: Constructing rockets. Difficulty escalates from building a single engine (Lv.1) to integrated aircraft (Lv.2) and finally to assembling separate substructures (engine and frame) into a launch-capable vehicle (Lv.3).

2. Language-Driven Construction & The Spatial Geometric Computation Library

To bridge the gap between natural language and physical simulation, the authors developed an open-source 3D Spatial Geometric Computation Library.

• Context: The benchmark utilizes the Besiege physics sandbox, which offers realistic physics but lacks a native language interface or API for programmatic control.
• Solution: The library faithfully reproduces Besiege's closed-source construction logic. It accepts actions from an LLM agent, computes state updates, and performs physical constraint checks (e.g., collision avoidance, face occupancy).
• Interaction: The LLM interacts via a text-based interface where actions are categorized into Build, Refine, Query, and Control. The library returns human-interpretable state descriptions or error explanations if constraints are violated.

3. LLM Agentic Workflow

The construction process is managed by a multi-agent workflow involving five collaborative entities (with a sixth, Controller, added for Transport tasks):

• Planner: Generates a high-level structured construction plan.
• Drafter: Creates detailed design schematics based on the plan.
• Reviewer: Verifies schematics for structural integrity and logical consistency, guiding revisions.
• Builder: Executes construction commands via the Spatial Geometric Computation Library.
• Guidance: Provides step-by-step execution suggestions and feedback analysis.
• Controller: Designs control configurations and sequences for dynamic tasks.

This workflow follows a coarse-to-fine structure and a multi-party debate framework to iteratively improve construction quality.

4. Evaluation

Evaluation is simulation-based using Besiege. For each task-LLM pair, 64 trials are run to ensure statistical reliability. Metrics include:

• Performance: Success rate, number of parts (lower is better for maintainability), and task-specific indicators (e.g., maximum transport distance, maximum load weight, thrust-to-weight ratio, maximum height).
• Cost: Input/output token counts and total LLM requests.

Key Results

The authors evaluated nine frontier LLMs (including GPT-5, GPT-4o, Claude-4, Grok-4, Gemini-2.0, DeepSeek-3.1, Qwen-3, Kimi-K2, and Seed-1.6).

• General Capability: Current LLMs demonstrate elementary construction capabilities. They can adapt to magnitude and ambiguity challenges (e.g., scaling up components for larger cargo) and occasionally propose creative, unconventional solutions (e.g., propulsion-powered vehicles).
• Limitations:
  • Hierarchical Assembly: Success rates drop sharply as assembly complexity increases (e.g., Support tasks with multiple substructures), indicating weak compositional reasoning.
  • Precision: In high-precision tasks like Lift, most models (except GPT-5 and Grok-4) fail to achieve success rates above zero at higher difficulty levels.
  • Spatial Reasoning: The most common failure modes are Overlap Conflict and Face Occupied, suggesting LLMs struggle to maintain accurate mental models of updated spatial structures.
• Cost vs. Performance: Massive inference (high token usage) does not guarantee high performance. Many failed attempts incurred massive token costs, while successful constructions often required moderate resources.
• Top Performers: GPT-5 and Grok-4 showed competitive performance across tasks, aligning with their general reasoning capabilities.

Significance and Claims

The paper claims to take the first step towards engineering automation using LLMs by introducing a benchmark that moves beyond static text or abstract planning to interactive, physics-grounded 3D construction.

• Technical Contribution: The work provides an extensible task design strategy and a novel open-source library that enables language-driven interaction with a closed-source physics engine, ensuring consistency between language-space actions and physical-world outcomes.
• Benchmarking Value: BuildArena serves as a diagnostic tool, revealing that while LLMs possess broad domain knowledge, they currently lack the robust spatial reasoning and precision required for complex engineering assembly.
• Future Direction: The authors note that while their default setting uses single-shot evaluation, preliminary closed-loop feedback experiments showed significant performance gains, suggesting that iterative refinement driven by simulation feedback is a viable path for future engineering automation.

The authors remain modest, acknowledging limitations such as the single-shot experimental setting and the restricted diversity of the basic module library, which currently constrains the range of constructible objects.