Technical Summary: GeoBuildBench

Problem Statement

While Large Language Models (LLMs) and Multimodal LLMs (MLLMs) have demonstrated strong performance on geometry benchmarks like GeoQA and Geometry3K, these evaluations primarily focus on answer correctness or static diagram interpretation. Existing approaches often assume formalized representations or pre-existing diagrams, leaving a fundamental gap: the ability of models to transform informal, natural-language descriptions into consistent, executable geometric structures.

Current "text-to-diagram" methods often rely on visual similarity metrics or parser-dependent checks that fail to verify whether geometric relations (e.g., parallelism, tangency) are semantically satisfied. Furthermore, agentic frameworks tested in textual environments have not been rigorously evaluated in domains where every action must satisfy strict spatial constraints, and failures (such as intersecting parallel lines) are immediately observable. The paper posits that geometry provides a uniquely stringent testbed for grounded reasoning, where errors in construction lead to visibly invalid diagrams, making it ideal for studying structural hallucinations.

Methodology

The GeoBuildBench Environment

The authors introduce GeoBuildBench, an interactive benchmark designed to evaluate agents that translate natural-language plane geometry problems into executable constructions. The environment operates on a closed agent–environment loop:

1. Task Definition: Each task consists of a natural-language problem, a set of required geometric objects, and a list of verification conditions. The benchmark does not assume a single "gold" diagram; multiple constructions are valid if they are executable and semantically accurate.
2. Domain-Specific Language (DSL): A compact, execution-oriented geometry DSL is defined. It is a construction language rather than a declarative constraint language. Agents must explicitly construct objects (points, lines, circles) and relations (intersections, parallels) rather than asserting them. The DSL supports primitives for geometric entities and operators for common constructions.
3. Execution and Verification: Generated DSL programs are executed by a geometry kernel. A verification module assesses the result across three dimensions:
   • Executability: Does the program run without errors?
   • Object Coverage: Are all required objects present?
   • Constraint Satisfaction: Are verifiable conditions (incidence, parallelism, metric constraints, angles) met?
4. Interaction Protocol: Agents operate in a bounded iterative setting (maximum 5 steps). They receive visual renderings and structured feedback regarding errors, missing objects, or violated constraints, allowing for iterative repair.

Dataset Curation

The benchmark comprises 489 Chinese textbook-style geometry problems curated from GeoQA and online repositories. To ensure solvability from text alone, a three-stage curation pipeline was employed:

1. Text-based Filtering: LLMs (GPT-4.1) filter out problems dependent on diagram-only cues, undefined references, or incomplete constraints.
2. Task Representation Extraction: A second LLM pass extracts structured task representations (required objects and verification conditions) into a fixed schema.
3. Human Verification: Graduate-level mathematics annotators verify schema conformity and geometric realizability, ensuring that a valid construction exists in principle.

Evaluation Setup

The authors evaluated four state-of-the-art multimodal models (GPT-5.1, Gemini-3-Flash, Qwen3-VL-235B, LLaMA-3.2-90B-Vision) using zero-shot or few-shot prompting. Models were evaluated on success rate, iteration steps, hallucination frequency, missing objects, and failed constraints.

Key Contributions

1. An Iterative Environment: A novel executable geometry construction environment with a minimal DSL and deterministic rendering, enabling grounded agent interaction and feedback loops.
2. The GeoBuildBench Dataset: A curated benchmark of 489 problems annotated with constraints to support evaluation without assuming a single gold construction, allowing for auxiliary constructions (e.g., helper points) as in human practice.
3. Empirical Analysis: A comprehensive evaluation revealing that while current MLLMs perform well on traditional geometry QA, they struggle with structural correctness and iterative self-repair in executable construction tasks.

Results

The evaluation highlights significant disparities in model performance:

• Success Rates: Frontier models GPT-5.1 (78.9%) and Gemini-3-Flash (75.3%) achieved the highest success rates. In contrast, Qwen3-VL-235B (42.2%) and LLaMA-3.2-90B-Vision (21.3%) performed significantly worse.
• Structural Hallucinations: A primary failure mode across all models was the "undefined reference," where agents referenced geometric objects not yet instantiated. Weaker models exhibited hallucination rates over two per problem, compared to ~0.3–0.4 for frontier models.
• Feedback Utilization: Frontier models demonstrated faster recovery from errors (Gemini-3-Flash: 1.17 steps/hallucination) compared to weaker models (~1.75 steps).
• Vision Ablation: Enabling vision inputs produced mixed results. For Qwen3-VL, vision increased success slightly but destabilized symbol binding (increasing missing objects and hallucinations). For LLaMA-3.2, vision reduced structural errors but did not improve overall success, indicating that geometric planning remains the dominant bottleneck.
• Auxiliary Constructions: Successful models frequently utilized auxiliary constructions (e.g., helper circles) to satisfy constraints, whereas weaker models tended to omit required relations or fail to construct necessary geometric support.

Significance and Claims

The paper claims that GeoBuildBench serves as a rigorous testbed for grounded, executable reasoning that goes beyond textual plausibility or visual similarity. The results suggest that:

• Current MLLMs, despite strong performance on standard geometry QA, struggle with maintaining a coherent geometric state and respecting relational constraints during multi-step construction.
• Many failure modes (structural hallucinations, infeasible constructions) would be missed by evaluation protocols relying solely on visual plausibility or answer correctness.
• Effective geometry construction requires sensitivity to explicit structural feedback and the ability to update internal representations accordingly, rather than just raw model size or general vision-language capacity.

The authors conclude that GeoBuildBench offers a controlled testbed for diagnosing these limitations and supports future work on structurally grounded, agentic reasoning for geometry. They acknowledge limitations, noting that the benchmark is constrained to Chinese plane geometry and a specific set of condition types, and that performance may vary under different interaction protocols or tooling.