Technical Summary: GIQ – Benchmarking 3D Geometric Reasoning of Vision Foundation Models

Problem Statement

Despite achieving state-of-the-art performance on standard visual benchmarks, modern monocular 3D reconstruction methods and vision-language models (VLMs) exhibit significant doubts regarding their true understanding of geometric properties. Current models often exploit arbitrary, 3D-unaware patterns from training datasets, leading to poor generalization on out-of-distribution data. Specifically, existing large-scale 3D datasets (e.g., OmniObject3D, NAVI, Google Scanned Objects) lack the precise annotations required to evaluate core 3D object geometry properties such as symmetry, convexity, and topological complexity. Consequently, there is a critical gap in understanding the "geometric intelligence" of foundation models: do they truly reason about 3D structure, or do they merely approximate it based on texture and 2D cues?

Methodology

To address these gaps, the authors introduce GIQ (Geometric IQ Test), a comprehensive benchmark dataset and evaluation framework designed to probe the geometric reasoning capabilities of vision and vision-language foundation models.

The GIQ Dataset

GIQ consists of 224 unique polyhedral shapes categorized into groups with varying levels of geometric complexity and symmetry:

• Classes: Platonic (5), Archimedean (13), Catalan (13), Johnson (92), Stellations (48), Kepler-Poinsot (4), Compounds (10), and Uniform non-convex (53).
• Data Modalities:
  • Synthetic: Rendered using the Mitsuba Physically Based Renderer with controlled lighting, diffuse shading, and 20 random viewpoints per shape to ensure diverse perspective coverage.
  • Real-World ("Wild"): Physical paper models constructed from geometric blueprints, photographed in diverse indoor and outdoor environments (varying lighting, backgrounds, and weather) to test environmental robustness.

Evaluation Tasks

The authors conducted systematic experiments across four distinct dimensions of geometric intelligence:

1. Monocular 3D Reconstruction: Evaluating state-of-the-art methods (Shap-E, Stable Fast 3D, OpenLRM) on reconstructing complete 3D geometries from single images. Models were tested zero-shot on GIQ.
2. 3D Symmetry Detection: Using linear and non-linear probes on image encoder embeddings (including CLIP, DINOv2, SigLip, and multi-view networks like DUSt3R) to classify specific 3D symmetry elements (central point reflection, 4-fold, and 5-fold rotational symmetries).
3. Mental Rotation Tests (MRT): Assessing spatial reasoning by determining if two images (one synthetic, one real) depict the same polyhedron under different rotations. This included a "hard split" with visually and geometrically similar shape pairs.
4. Zero-Shot Polyhedron Classification: Querying frontier VLMs (ChatGPT o3/o4, Gemini 2.5 Pro, Claude 3.7) to identify polyhedron names from images, testing their ability to distinguish face geometry, convexity, and compound structures.

Key Results

1. Monocular 3D Reconstruction Failures

Even models trained on millions of diverse 3D assets struggled significantly:

• Basic Shapes: State-of-the-art algorithms frequently failed to reconstruct basic Platonic solids (e.g., cubes) accurately, producing asymmetric artifacts and mismatched global structures.
• Complexity Degradation: Performance degraded further with non-convex or compound forms, exhibiting fractured surfaces and topological inconsistencies.
• Domain Gap: Models trained on synthetic data or specific datasets failed to generalize to "wild" images, often failing to recover fundamental properties like axis-aligned faces or right angles.

2. Symmetry Detection vs. Geometric Differentiation

• Symmetry: Linear probing on foundation image encoders (particularly DINOv2) showed strong capability in detecting 3D symmetry elements, achieving up to 93% accuracy on wild images for 4-fold rotational symmetry.
• Differentiation: Despite capturing symmetry cues, these same encoders and VLMs faltered significantly in tasks requiring detailed geometric differentiation, such as the Mental Rotation Test. In hard splits, most models approached chance levels, with only non-linear probes on SigLIP and DINOv2 reaching ~67-69% accuracy.

3. Human vs. Machine Performance

• Mental Rotation: While the best model configuration (SigLIP with non-linear probe) matched the average human accuracy (~69%), 68% of individual human participants outperformed the best model. Top human performers achieved 90% accuracy on hard sets.
• Zero-Shot Classification: Frontier VLMs exhibited remarkably low accuracy (<20%) on Catalan, Johnson, and non-convex shapes. They frequently hallucinated face types, confused convexity (e.g., misclassifying concave shapes as convex), and conflated compound structures with stellations.
• Geometry-Native Models: Models designed for 3D inputs (LLaVA-3D, ShapeLLM, PointLLM) did not outperform generalist VLMs, suggesting a fundamental architectural gap in explicit geometric representation.

Significance and Claims

The paper positions GIQ as a "geometric litmus test" for spatial intelligence. The authors claim that the results reveal a substantial disconnect between the impressive capabilities of foundation models on standard benchmarks and their performance in tasks requiring explicit geometric reasoning.

• Implicit vs. Explicit Understanding: The study suggests that while foundation models may implicitly encode 3D structural properties (evidenced by symmetry detection), this does not translate to robust, explicit reasoning. Models appear to learn priors for noisy, imperfect surfaces from training data rather than mathematical exactness.
• Limitations of Scale: The authors argue that scaling alone is insufficient; without mathematically generated data to enforce geometric constraints, models fail to grasp the "geometric arithmetic" necessary for complex scenes.
• Future Direction: GIQ provides a structured ontology to diagnose these gaps and facilitate the development of robust, geometry-aware representation learning. The authors propose that the dataset's structure enables the automatic generation of precise Chain-of-Thought training data, though they note that future work must address generalization to broader wild distributions.

In conclusion, the paper asserts that current vision systems rely on texture-based approximations rather than true geometric understanding, leaving them unprepared for the "geometric calculus" of complex real-world scenarios. GIQ serves as a critical diagnostic tool to drive the development of models capable of human-level spatial reasoning.