🤖 machine learning

Toward an Artificial General Teacher: Procedural Geometry Data Generation and Visual Grounding with Vision-Language Models

This paper addresses the failure of existing vision-language models on geometric diagrams by introducing a fully automated procedural data engine that generates 200,000 synthetic training examples, enabling a fine-tuned model to achieve high-accuracy visual grounding and establishing a foundation for Artificial General Teachers in geometry education.

Original authors: Hai Nguyen-Truong, Alper Balbay, Tunga Bayrak

Published 2026-04-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hai Nguyen-Truong, Alper Balbay, Tunga Bayrak

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to be a geometry teacher. You want it to look at a drawing of a triangle, hear you say, "Show me the height of this triangle," and then immediately point its digital finger exactly at the right line on the paper.

This is exactly what the researchers in this paper tried to build: an Artificial General Teacher (AGT). But they hit a massive wall. Here is the story of how they fixed it, explained simply.

The Problem: The "Photo vs. Sketch" Confusion

Imagine you teach a dog to recognize a "ball" by showing it thousands of photos of red, bouncy, fuzzy balls in parks. The dog learns perfectly. Then, you show the dog a simple black-and-white line drawing of a ball on a piece of paper. The dog is confused. It doesn't see the color or the texture; it just sees a circle.

This is what happened to existing AI models.

The Old Models: They were trained on photos of real life (people, dogs, cars) where things have colors, textures, and shadows.
The Geometry World: Geometry diagrams are the opposite. They are abstract, black lines on a white background. They have no texture.
The Result: When the researchers asked these "photo-trained" AI models to point at a triangle in a math diagram, they failed miserably. They got it right less than 1% of the time. It was like asking a dog to find a ball in a sketch; the dog just guessed randomly.

The Solution: Building a "Magic Factory"

Since there were no existing textbooks with thousands of labeled geometry diagrams for the AI to study, the researchers had to build their own. But they didn't want to hire humans to draw and label them (that takes too long).

Instead, they built a Procedural Data Engine. Think of this as a 3D printer for math problems.

The Blueprint: They wrote a computer program that understands the rules of geometry (like "a rectangle must have 90-degree corners").
The Assembly: The program randomly picks rules to build millions of unique shapes (squares, trapezoids, triangles) automatically.
The Labeling: Because the computer built the shape, it knows exactly where every line is. It automatically creates a "mask" (a digital sticker) that covers exactly the right part of the drawing.
The Speech: The program also writes different ways to ask for the shape, like "the line from A to B" or "the top side of the triangle."

In just 12 hours, this factory produced 200,000+ perfect practice problems without a single human drawing a line.

The Training: Teaching the Robot to "Read"

Once they had the data, they needed to teach the AI. They didn't just throw the data at the AI; they used a smart technique called Fine-Tuning.

The Student: They picked two powerful AI models (Florence-2 and Qwen-VL). Think of them as smart students who already know how to read and look at pictures, but they've never studied geometry.
The Lesson: They showed the AI the 200,000 synthetic diagrams.
The Trick: Instead of asking the AI to guess every single pixel (which is hard and slow), they taught it to speak in coordinates. It's like telling the AI, "Don't draw the whole triangle; just give me the list of points that make up the corners." This made the AI much faster and more accurate.

The Result: From Clueless to Expert

The results were dramatic:

Before Training: The AI was guessing randomly (<1% accuracy).
After Training: The AI got it right 49% of the time using standard measures, and 85% of the time using a new, smarter way of measuring.

What is this "New Way" (Buffered IoU)?
Imagine you are drawing a very thin line on a piece of paper. If you are off by just one tiny pixel, a standard computer might say, "You failed! That's a different line!"
The researchers invented a Buffered Score. It's like giving the AI a "fuzzy marker." If the AI's line is very close to the real line (within a tiny margin), the score says, "Good job, you got the right shape!" This is much fairer for thin geometry lines.

Why Does This Matter?

This isn't just about drawing lines. This is a stepping stone to an Artificial General Teacher.

Imagine a future where you have a tutor that can:

Read your math problem.
Solve it step-by-step.
Point directly at the part of the diagram it is talking about while it explains.

This paper solved the "pointing" part. By teaching the AI to understand that a math diagram is different from a photo, and by building a factory to create endless practice tests, they have laid the foundation for a robot teacher that can actually see what it's teaching.

In a nutshell: They built a robot factory to make perfect math practice sheets, taught a smart AI to read those sheets instead of photos, and now the AI can finally point at the right lines in a geometry problem.

1. Problem Statement

The paper addresses the challenge of Visual Grounding in Geometry Education, specifically formalized as a Referring Image Segmentation (RIS) task.

The Task: Given a geometric diagram and a natural language description (e.g., "the altitude of triangle ABC"), the model must produce a pixel-level segmentation mask highlighting the referred element.
The Core Challenge (Domain Shift): State-of-the-art RIS models trained on natural image benchmarks (e.g., RefCOCO) fail catastrophically on geometric diagrams.
- Natural Images: Rich textures, colors, and contextual cues define objects.
- Geometric Diagrams: Sparse black lines on white backgrounds; semantic identity relies on topological relationships and text labels rather than visual appearance.
- Result: Zero-shot models achieve <1% Intersection-over-Union (IoU) on geometry tasks, effectively performing randomly.
Data Scarcity: There is a lack of large-scale, annotated datasets for geometric diagrams with pixel-perfect masks, hindering the training of specialized models.

2. Methodology

The authors propose a three-part solution: a fully automated data generation engine, domain-specific model fine-tuning, and a geometry-aware evaluation metric.

A. Procedural Data Engine (Zero Manual Annotation)

The authors developed a pipeline to generate 200,000+ synthetic geometry diagrams with perfect ground truth.

Constraint-Based Generation: Instead of random sampling, the system analytically solves geometric constraints (e.g., parallelism, angle bisectors) to generate valid shapes (7 quadrilateral types, including trapezoids, rhombuses, and shapes with incircles).
Vector Rendering: Shapes are rendered using LaTeX/TikZ templates. This ensures resolution independence, determinism, and high-quality typography.
Mask Extraction: The system uses the LaTeX compiler itself to generate ground-truth masks. By rendering the target element in red and others in black, then applying channel arithmetic ( $R - G - B$ ), it extracts binary masks that are pixel-perfect and perfectly aligned with the input image, including anti-aliasing.
Language Generation: A hierarchical template engine creates diverse referring expressions (from direct labels like "line AB" to complex topological descriptions) to prevent phrase memorization.
Stochastic Resolution: To improve robustness, 20% of data is rendered at low DPI (72–150) to simulate real-world scanning artifacts.

B. Domain-Specific VLM Fine-Tuning

The authors fine-tuned two Vision-Language Models (VLMs) using LoRA (Low-Rank Adaptation):

Models:
- Florence-2 (230M params): A unified foundation model optimized for grounding.
- Qwen-VL (7B params): A large model with a strong LLM backbone for complex reasoning.
Architecture Strategy:
- Polygon-Based Representation: Instead of predicting dense pixel masks (inefficient for thin lines), the model autoregressively generates coordinate tokens (e.g., <seg>x1, y1... </seg>). This reduces sequence length by 5–10× compared to Run-Length Encoding (RLE).
- Training: The vision encoder remains frozen; LoRA adapters are inserted into attention layers to learn task-specific coordinate generation.
- Thin Structure Handling: Training masks are morphologically dilated to relax localization requirements, and post-processing (thinning) is applied at inference.

C. Geometry-Aware Evaluation (Buffered IoU)

Standard IoU is too harsh for thin geometric structures (1–3 pixel lines), where a 1-pixel shift can drop the score to near zero.

Metric: Buffered IoU (BIoU).
Mechanism: Both the predicted and ground-truth polygons are expanded by a buffer distance ( $\beta$ , typically 3 pixels) using Minkowski sums before calculating IoU.
Benefit: This metric tolerates minor spatial offsets while rewarding topological correctness, providing a more faithful measure of segmentation quality for diagrams.

3. Key Contributions

Automated Data Engine: A fully automated pipeline generating 50,000+ (scalable to 200k+) diagram-mask-description triplets with pixel-perfect ground truth, requiring zero human annotation.
Domain Adaptation: Demonstrated that fine-tuning VLMs on synthetic geometry data bridges the domain gap, transforming performance from <1% to high accuracy.
New Evaluation Metric: Introduced Buffered IoU (BIoU), which better reflects the quality of thin-structure segmentation than standard IoU.
Efficient Training: Showed that parameter-efficient fine-tuning (LoRA) on compact models (Florence-2) can outperform larger models (Qwen-VL) for this specific task, with training completed in hours on standard GPUs.

4. Experimental Results

Zero-Shot Failure: All zero-shot models (including LAVT, Florence-2, Qwen-VL) achieved <1% IoU on the geometry test set, confirming the severity of the domain shift.
Fine-Tuned Performance:
- Florence-2 (Fine-tuned): Achieved 49% IoU and 85% BIoU.
- Qwen-VL (Fine-tuned): Achieved 10% IoU and 42% BIoU (suggesting it requires further hyperparameter tuning).
- Insight: Despite being 30× smaller, Florence-2 significantly outperformed Qwen-VL, likely due to its architecture being more suited for grounding tasks.
Robustness: Mixed-resolution training improved low-DPI performance by 24 percentage points in BIoU without degrading high-DPI results.
Generalization: The model showed robustness across varying description complexities (short vs. long phrases), indicating genuine visual-linguistic grounding rather than keyword matching.

5. Significance and Future Outlook

Artificial General Teachers (AGTs): This work establishes a critical component for AGTs—systems capable of not just solving math problems but visually explaining them. By "pointing" to specific parts of a diagram, an AGT can reduce cognitive load and anchor abstract reasoning in concrete visuals.
Paradigm Shift: The paper proves that for domains requiring mathematical precision (like geometry), procedural generation is superior to manual annotation. It leverages the exactness of mathematical constraints to create perfect training data.
Limitations & Future Work:
- Current data is synthetic; future work needs Sim-to-Real transfer for handwritten or scanned diagrams.
- Scope is limited to 2D quadrilaterals; future extensions could include 3D geometry and implicit constructions (e.g., altitudes not explicitly drawn).
- The system currently handles single-element queries; multi-element segmentation is a future goal.

In conclusion, this paper provides a foundational framework for building AI tutors that can visually ground geometric concepts, solving the data scarcity problem through procedural generation and overcoming domain shifts through targeted fine-tuning and novel evaluation metrics.