🤖 AI

2.5-D Decomposition for LLM-Based Spatial Construction

This paper introduces a neuro-symbolic pipeline utilizing 2.5-D decomposition to significantly improve LLM-based spatial construction accuracy by restricting the model to 2D planning while a deterministic executor handles vertical placements, achieving near-ceiling performance on benchmarks and demonstrating successful transfer to edge hardware and diverse collaborative tasks.

Original authors: Paul Whitten, Li-Jen Chen, Sharath Baddam

Published 2026-05-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Paul Whitten, Li-Jen Chen, Sharath Baddam

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, but slightly clumsy, robot how to build a tower out of colorful blocks based on a spoken instruction like, "Build a T-shape and add a purple block to the top."

The problem is that while the robot's brain (a Large Language Model, or LLM) is great at understanding the idea of the shape, it is terrible at doing the math for where to put the blocks in 3D space. It often forgets that gravity exists, stacking blocks in mid-air or miscounting how high a stack should be.

This paper presents a clever solution called 2.5-D Decomposition. Here is how it works, using simple analogies:

The Problem: The "Clumsy Architect"

Think of the LLM as a brilliant architect who can draw a perfect 2D blueprint on a piece of paper. It knows exactly where the walls should go on the floor plan. However, if you ask this architect to also calculate the exact height of every single brick in a 3D tower, it starts making mistakes. It might say, "Put a brick here," without realizing there's already a brick underneath it, or it might stack them too high.

The Solution: The "Gravity-Proof Foreman"

The authors realized that in many building tasks, the height of a block isn't a free choice; it's determined by physics. If you are building on a flat floor with gravity, a new block must sit on top of the highest block already there. The height is automatic.

So, they split the job into two parts:

The Architect (The LLM): This part is only allowed to draw the 2D floor plan. It says, "Put a red block at position X, and a blue block at position Z." It is forbidden from talking about height (Y). It only plans the "footprint" of the building.
The Foreman (The Deterministic Code): This is a simple, unthinking computer program. Its only job is to look at the floor plan and say, "Okay, you want a block at X and Z? Well, gravity says it has to go on top of whatever is already there." It calculates the height automatically.

By taking the "height" decision away from the clumsy architect and giving it to the unthinking foreman, they eliminated a huge category of mistakes.

The "Peephole" Trick

The paper also mentions a "Peephole Prompt Optimization." Imagine the architect is prone to specific, predictable daydreams. For example, if you say "extend the end," the architect might accidentally build a tower instead of extending the floor.

The researchers created a "spot-check" system. Before the architect draws the plan, a quick scanner looks at the instructions. If it sees a phrase known to cause trouble (like "each end"), it slips a tiny, specific note into the architect's ear: "Hey, when you see 'each end,' remember to extend sideways, not upwards." This acts like a safety net for the architect's known weak spots.

The Results

When they tested this system:

The Old Way: Using just the LLM to do everything (3D planning) resulted in about 76% to 90% accuracy.
The New Way: Using the 2.5-D split (Architect + Foreman) pushed accuracy up to 94.6%.
The Surprise: A smaller, cheaper AI model (GPT-4o-mini) using this new method actually beat a much larger, more expensive model (GPT-4o) that didn't use the method.

Real-World Portability

The paper also showed that this trick isn't just for one specific computer. They ran the same system on a small, powerful computer chip (an NVIDIA Jetson Thor) that could sit on a robot's head, and it worked just as well as the giant cloud computers.

The Bottom Line

The main takeaway is simple: Don't ask a language model to do math it's bad at. If a part of a task is fixed by the laws of physics (like gravity determining height), let a simple computer program handle that part. Let the AI focus only on the creative, flexible parts of the plan. This makes the whole system much more reliable, even if the AI itself isn't perfect.

Note: The paper mentions that about 5% of the remaining errors came from a different "Architect Agent" that answers clarification questions, which is a separate limitation they couldn't fix with this specific method.

Technical Summary: 2.5-D Decomposition for LLM-Based Spatial Construction

Problem Statement

Autonomous systems tasked with constructing physical structures from natural language instructions face a critical reliability gap: while Large Language Models (LLMs) excel at understanding high-level intent, they systematically fail at precise spatial reasoning, particularly regarding three-dimensional coordinate generation. Previous benchmarks indicate that LLMs frequently produce errors in vertical placement, off-by-one stacking, and duplicate positions. These failures stem from a lack of internal world models to enforce physical constraints, such as gravity, which dictate that a block's vertical position is strictly determined by the occupancy of the column beneath it.

Methodology

The paper proposes a neuro-symbolic pipeline based on 2.5-D decomposition to address these coordinate errors. The core insight is that in gravity-constrained construction domains, the vertical axis ( $y$ ) is not a free variable but a deterministic function of the horizontal position ( $x, z$ ) and the current state of the grid.

The 2.5-D Decomposition Pipeline

The system separates the planning and execution tasks into distinct modules:

LLM Planner (2D): The LLM is restricted to generating a plan in the two-dimensional horizontal plane. It specifies only $(x, z)$ coordinates, block colors, action types, and counts. It is explicitly prevented from outputting vertical ( $y$ ) coordinates.
Deterministic Executor (3D): A symbolic engine computes the vertical placement for every block based on column occupancy. The vertical coordinate $y^*$ for a new block at $(x, z)$ is calculated as the minimum height where the cell is empty, ensuring gravity constraints are strictly met:
$y^*(x, z, G) = \min\{y \in \{0, \dots, 4\} \mid (x, y, z) \notin \text{dom}(G)\}$
This eliminates an entire class of coordinate errors by removing the $y$ -dimension from the LLM's output space.

Supporting Components

The pipeline includes four additional mechanisms to handle ambiguity and systematic failure modes:

Structure Analyzer: Detects geometric primitives (e.g., T-shapes, L-shapes) in the existing grid to inform the planner.
Underspecification Detector: Uses a decision-theoretic framework to determine when to ask clarification questions. It calculates the expected value of guessing versus asking, prioritizing color inquiries over count inquiries when ambiguity exists.
Peephole Prompt Optimizer: Scans instructions for specific trigger phrases (e.g., "each end," "in front of") known to cause systematic LLM errors. When detected, it injects targeted, pattern-matched corrections and worked examples into the prompt before the LLM call.
Plan Verifier: A rule-based module that validates the generated plan against the instruction and grid state, performing corrections for direction consistency and stacking plausibility.

Key Contributions

2.5-D Decomposition: A novel architectural pattern that restricts LLM planning to a 2D horizontal plane while delegating vertical placement to a deterministic executor, effectively eliminating vertical coordinate errors.
Peephole Prompt Optimization: A methodology for identifying systematic LLM failure modes triggered by specific input patterns and correcting them via dynamic prompt injection, analogous to compiler optimization.
Empirical Validation: Demonstration that this pipeline enables a smaller, cost-effective model (GPT-4o-mini) to outperform a larger, more expensive model (GPT-4o) in structural accuracy.
Decision-Theoretic Framework: A formal approach for handling underspecified instructions (missing colors or counts) by weighing the probability of a correct guess against the penalty of asking a clarification question.

Experimental Results

The system was evaluated on the Build What I Mean (BWIM) benchmark, consisting of 160 rounds of block construction tasks.

Accuracy: The full pipeline using GPT-4o-mini achieved a mean structural accuracy of 94.6% across 12 independent runs.
Comparison: This result outperforms the best competing system (using GPT-4o without decomposition) at 76.3% and the direct GPT-4o baseline at 90.3%.
Ablation Study: Removing the 2.5-D decomposition caused the most significant performance drop, reducing accuracy by 50.7 percentage points (to 43.8%), confirming it as the dominant contributor to success. Underspecification handling and peephole optimization also contributed significantly (17.9 pp and 11.3 pp, respectively).
Edge Deployment: The pipeline transferred successfully to local edge hardware. Nemotron-3 120B, running locally on an NVIDIA Jetson Thor AGX, achieved 94.5% accuracy with no prompt modifications, matching the cloud-based GPT-4o-mini results.
Generalization: A transfer experiment on 500 tasks from the IGLU collaborative building dataset showed that the 2.5-D decomposition alone improved block-level F1 scores from 0.723 to 0.798, confirming the approach's applicability beyond the primary benchmark.

Significance and Claims

The paper argues that the primary limitation in LLM-based spatial construction is not a lack of reasoning capability, but the inclusion of deterministic dimensions in the model's output space. By identifying dimensions that are computable from physical constraints (such as gravity) and removing them from the LLM's responsibility, the system achieves reliability comparable to the theoretical ceiling imposed by the quality of the clarification agent (the "architect").

The authors claim that this 2.5-D structure is a generalizable principle for any autonomous system where physical laws constrain assembly or placement. Examples cited include terrain-following UAV path planning (where altitude is a function of horizontal position and terrain data) and automated equipment maintenance (where part insertion paths are constrained by machine geometry). The work suggests that reliable LLM-guided operation in physically constrained environments is achievable not by training larger models, but by architecting systems that offload deterministic reasoning to symbolic code.

The authors note a hard ceiling on achievable accuracy: approximately 97.6%, bounded by errors in the architect agent (an LLM that answers clarification questions). The current system's 94.6% accuracy leaves only a 3.0 percentage point margin for builder-side improvements, indicating that the proposed pipeline has nearly maximized the potential of the builder component.