💬 NLP

Measuring Representation Robustness in Large Language Models for Geometry

This paper introduces GeoRepEval, a framework demonstrating that large language models exhibit significant performance gaps and representation-specific failures in geometry reasoning—particularly with vector formulations—revealing that current models rely on surface-level heuristics rather than robust, abstract geometric understanding.

Original authors: Vedant Jawandhia, Yash Sinha, Murari Mandal, Ankan Pal, Dhruv Kumar

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Vedant Jawandhia, Yash Sinha, Murari Mandal, Ankan Pal, Dhruv Kumar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant student who is taking a math test. You ask them, "What is the area of a triangle with a base of 5 and a height of 12?" They confidently write down "30." You ask again, but this time you describe the triangle using a grid map (coordinates), and they still get "30." So far, so good.

But then, you ask the same question using a third method: describing the triangle using arrows and directions (vectors). Suddenly, the student gets confused, makes a calculation error, and gives the wrong answer.

This is exactly what the paper "Measuring Representation Robustness in Large Language Models for Geometry" discovers about modern AI.

Here is the breakdown of the research in simple terms:

1. The Core Problem: The "Translation" Trap

The researchers found that Large Language Models (LLMs) are like students who are great at memorizing specific phrases but terrible at understanding the concept behind them.

In geometry, the same problem can be written in three different "languages":

Euclidean: The classic way ("Draw a triangle with these angles...").
Coordinate: Using an X and Y graph ("Point A is at (0,0), Point B is at...").
Vector: Using arrows and direction ("Move 3 units right, 4 units up...").

The paper argues that if an AI is truly smart, it shouldn't matter which "language" you use. The answer should be the same. But the AI failed this test. It was like a chef who can cook a perfect steak if you say "grill it," but burns it if you say "sear it," even though it's the exact same piece of meat.

2. The Experiment: The "GeoRepEval" Test

The team built a special testing ground called GeoRepEval.

They took 158 real geometry problems from high school textbooks.
They translated every single problem into all three "languages" (Euclidean, Coordinate, Vector).
They asked 11 different AI models (including big names like GPT, Claude, and LLaMA) to solve all of them.

The Result: The AI models were inconsistent.

When asked in the "Euclidean" language, they were often correct.
When asked in the "Vector" language, their accuracy dropped significantly—sometimes by 14 percentage points.
Some models were so fragile that they only got the answer right in all three languages less than 5% of the time.

3. The "Weak Link" Analogy

The researchers introduced a metric called Invariance@3. Think of it like a chain with three links.

If the AI gets the answer right in Euclidean, Coordinate, and Vector, the chain is strong.
If it fails even one of them, the chain breaks.

They found that for most models, the Vector link was the weak one. It was the "Achilles' heel." The models could handle the other two formats fine, but the moment they saw arrows and vector notation, they stumbled.

4. Why Does This Happen? (The "Why" Behind the "What")

The paper suggests a few reasons why the AI struggles with vectors:

The "Recipe" Length: Solving a problem with vectors often requires a longer, more complex chain of algebraic steps. It's like asking the AI to follow a recipe with 20 steps instead of 5. The more steps, the more likely it is to drop an ingredient (make a math error).
Training Bias: The AI was likely trained on more internet text about standard geometry (Euclidean) and less about advanced vector math. It's like a student who has read 1,000 books about baseball but only 10 about cricket; they will play baseball better, even if the rules are similar.
Fragile Math: The models seem to rely on "templates" they've seen before. When the format changes to vectors, the template doesn't fit, and the model tries to force it, leading to errors.

5. The Magic Fix: "Translate First"

The most interesting part of the paper is the solution they tested. They tried a trick called "Convert-Then-Solve."

Instead of asking the AI to solve the vector problem directly, they asked it to:

First: "Translate this vector problem into a standard Euclidean problem."
Second: "Now, solve the Euclidean problem."

The Result: This simple trick was a game-changer for the smarter models.

For high-capacity models, accuracy on vector problems jumped by up to 52 percentage points.
This proved that the AI actually knew how to solve the problem; it just couldn't handle the "vector" format directly. It needed a translator.

However, for the smaller, weaker models, this trick didn't work. They still failed, suggesting they lack the fundamental reasoning ability, not just the translation skill.

The Big Takeaway

This paper teaches us that high scores on standard tests don't mean an AI is truly "smart."

If an AI can solve a math problem in one format but fails in another, it isn't doing deep reasoning; it's just recognizing patterns. It's like a parrot that can say "I love math" but doesn't actually understand math.

For the future, we need to test AI not just on if they get the right answer, but on how consistent they are when we change the way we ask the question. If we want AI to be reliable in the real world (where problems come in all shapes and sizes), we need to fix this "translation" weakness.

1. Problem Statement

While Large Language Models (LLMs) have shown increasing proficiency in mathematical reasoning, their robustness to equivalent problem representations remains poorly understood. In geometry, a single problem can be expressed in three distinct but semantically identical forms:

Euclidean: Verbal/relational descriptions (e.g., "Triangle ABC has sides...").
Coordinate: Cartesian plane formulations (e.g., "Points A(0,0), B(3,4)...").
Vector: Dot/cross-product operations (e.g., "Given vectors $\vec{u}$ and $\vec{v}$ ...").

Existing benchmarks typically evaluate models on fixed formats, implicitly assuming representation invariance. This masks failures where a model solves a problem in one format but fails in another due to reliance on representation-specific heuristics rather than abstract geometric reasoning. The paper addresses the gap in understanding whether LLMs possess true geometric reasoning or merely pattern-match specific syntactic structures.

2. Methodology: GeoRepEval Framework

The authors propose GeoRepEval, a representation-aware evaluation framework designed to measure correctness, invariance, and consistency across parallel problem formulations.

A. Dataset Construction

Source: 158 core high-school geometry problems curated from standard Indian textbooks (NCERT, RD Sharma, RS Aggarwal).
Parallel Generation: Each problem was instantiated into three mathematically equivalent variants (Euclidean, Coordinate, Vector) using Gemini Pro, followed by manual verification by mathematical experts to ensure semantic equivalence and answer uniqueness.
Scale: The final dataset comprises 474 instances (158 problems $\times$ 3 representations).

B. Evaluation Metrics

The framework introduces two novel higher-order metrics alongside standard accuracy:

Invariance@3: The fraction of problems solved correctly in all three representations simultaneously.
- Property: It is bounded by the weakest representation's accuracy ( $Invariance@3 \le \min(Acc_{euc}, Acc_{coord}, Acc_{vec})$ ).
Consistency@3: The fraction of problems where the model produces the identical output across all three representations (regardless of whether the answer is correct).
- Property: $Consistency@3 \ge Invariance@3$ . The gap between them indicates systematic errors (same wrong answer) rather than representation sensitivity.
Failure Pattern Analysis: Problems are categorized by correctness patterns (e.g., CCW: Correct on Euclidean/Coordinate, Wrong on Vector) to identify specific failure modes.

C. Statistical Rigor

Controls: Regression analyses control for surface features like token length and symbolic density.
Testing: Uses bootstrap confidence intervals (10,000 resamples) and paired McNemar tests to confirm statistical significance of observed gaps.

3. Key Contributions

GeoRepEval Framework: A statistically grounded evaluation pipeline with formally justified metrics linking invariance and consistency to standard accuracy.
Parallel Benchmark: The first dataset providing 158 core geometry problems in three semantically equivalent representations, enabling problem-level invariance testing.
Empirical Evidence of Fragility: Demonstrates that accuracy gaps of up to 14 percentage points exist solely due to representation choice, with vector formulations being the primary failure point.
Intervention Analysis: Introduces a "Convert-then-Solve" (CTS) prompting strategy, showing that high-capacity models can achieve near-perfect vector accuracy if first prompted to translate the problem into Euclidean form.

4. Key Results

The study evaluated 11 LLMs (ranging from 7B to 20B parameters, including open-weight and proprietary models like LLaMA, GPT, Claude, and Gemini).

Representation Bias: Euclidean formulations consistently yielded the highest accuracy, while Vector formulations were the dominant failure point.
- Example: Claude-Haiku achieved 60% accuracy on Euclidean but only 46% on Vector (14 pp gap).
- Low-End Performance: LLaMA-3.1-8B achieved an Invariance@3 of only 0.044, meaning fewer than 5% of problems were solved correctly across all three forms.
Failure Patterns: The CCW pattern (Correct on Euclidean/Coordinate, Wrong on Vector) was 2–5 times more frequent than WCC (Wrong on Euclidean/Coordinate, Correct on Vector). This suggests models apply Euclidean reasoning templates fluently but struggle with vector procedural chains.
Statistical Significance: The accuracy gaps between Euclidean and Vector representations were statistically significant ( $p < 0.05$ ) for 8 out of 11 models, even after controlling for prompt length and symbolic complexity.
Convert-then-Solve (CTS) Intervention:
- High/Mid-Capacity Models: CTS improved vector accuracy by up to 52 percentage points (e.g., Gemini-2.5-Flash went from 45% to 97%). This suggests their failures are due to representation sensitivity, not a lack of reasoning capability.
- Low-Capacity Models: Models like LLaMA-3.1-8B showed no significant gain with CTS, indicating deeper reasoning limitations that prompting cannot fix.

5. Significance and Implications

Heuristics vs. Reasoning: The results suggest current LLMs rely heavily on representation-specific heuristics (likely learned from training data distributions where Euclidean/Coordinate forms are more common) rather than abstract geometric reasoning.
Benchmark Limitations: Single-format benchmarks are insufficient for assessing true reasoning capabilities; a model can appear robust while failing on semantically identical variations.
Safety and Reliability: For safety-critical applications, the choice of representation materially affects reliability. GeoRepEval provides a tool for diagnosing these risks.
Future Directions: The paper highlights the need for training-time interventions to improve representation robustness and suggests extending this framework to other mathematical domains and reasoning strategies (e.g., Chain-of-Thought).

In conclusion, the paper establishes that representation invariance is a critical, unmet benchmark for LLMs. While high-capacity models possess the underlying reasoning ability, they fail to access it consistently across different mathematical formalisms without specific prompting interventions.