Math-Vision Diagrams: A Comprehensive Benchmark for Evaluating LLM Mathematical Diagram Generation Capabilities
This paper introduces "Math-Vision Diagrams," the first comprehensive benchmark designed to evaluate Large Language Models' capabilities in generating mathematically precise diagrams from textual prompts by unifying text-to-code and text-to-image paradigms, revealing significant current limitations in this critical skill.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are trying to teach a robot how to draw a perfect blueprint for a house. You wouldn't just say, "Draw a house," and hope it comes out with straight walls and a door in the right spot. You'd need to explain exactly where the corners go, how long the beams are, and that the roof must be a specific angle. This is the world of mathematical diagram generation. It sits at the intersection of two superpowers: mathematical reasoning (the brainy logic that knows a triangle's angles must add up to 180 degrees) and visual creation (the artistic ability to actually draw the picture). For a long time, computers have gotten really good at looking at pictures and answering questions about them, or at writing code to make simple charts. But asking a computer to look at a word problem and then draw the exact, mathematically correct diagram from scratch? That's like asking a chef to read a recipe and then instantly bake a cake that looks exactly like a photo, with the frosting perfectly piped and the layers even. It's a huge challenge because math doesn't allow for "close enough." If a line is off by a millimeter or a label is in the wrong place, the whole diagram becomes useless, like a map that leads you off a cliff.
This is exactly the problem a team of researchers from Pandita AI Inc. decided to tackle. They realized that while we have plenty of tests to see if robots can solve math problems using pictures, we didn't have a fair way to test if robots could create those pictures in the first place. So, they built a new playground called Math-Vision Diagrams. Think of it as a giant, rigorous art class for AI, where the teacher (the benchmark) gives the student (the AI) a text description and demands a drawing that is not just pretty, but mathematically perfect.
The researchers started by gathering a massive collection of 3,040 tricky math problems from real competitions, covering everything from geometry to statistics. They filtered this down to 2,920 problems that relied heavily on visual diagrams. Then, they used a clever mix of other AI models and human math experts to turn the original, sometimes vague, problem statements into crystal-clear instructions. For example, instead of just saying "draw a circle," the new prompts would say, "Draw a circle with a horizontal and vertical diameter intersecting at the center, labeled 'O'." They even had human experts double-check that these instructions were clear enough for a human to draw the picture perfectly, scoring them on a scale of 1 to 5.
Once they had their "exam questions" ready, they put 11 different AI models to the test. Some of these models are like architects who write detailed blueprints (code) that a computer then builds into an image (text-to-code). Others are like digital painters who try to paint the picture directly from the description without writing any code first (text-to-image). The researchers measured how well the drawings matched the original math problems using several tools: checking if the lines and edges lined up perfectly, how similar the overall look was to the target, and whether the code actually worked without crashing.
The results were a mix of impressive progress and some humbling reality checks. The study found that no single AI model is a master of everything. The "architect" models (the code generators) were generally better at getting the math and structure right—like making sure a square actually has four equal sides—but they often failed to finish the job because their code wouldn't compile, crashing about 10% to 30% of the time. The "painter" models (the direct image generators) were incredibly reliable, almost always producing an image, but their drawings often lacked the precise mathematical structure, with lines that were slightly wobbly or labels in the wrong spots.
One of the most interesting findings was that even the most advanced models, including some of the biggest names in AI, struggled significantly. The best-performing model, Claude Opus 4.6, managed to get the visual look and math structure pretty close, but it still made mistakes. Meanwhile, a model called GPT-5.4 performed surprisingly poorly, often creating overly complicated drawings that were mathematically wrong, suggesting that sometimes "thinking" too hard about the problem can actually confuse the drawing process. The researchers also noticed that while simple geometry was becoming manageable for these AIs, more complex topics like statistical charts or abstract topological shapes remained a major hurdle.
Ultimately, the paper suggests that we are still in the early days of this technology. While AI can now generate decent-looking diagrams, it hasn't quite mastered the "perfect precision" needed for serious math and science yet. The team has made all their data, code, and testing tools available for everyone to use, hoping that by shining a light on exactly where these models fail, we can help build the next generation of AI that can finally draw a perfect circle, every single time.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.