Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams
This paper introduces Diagram-MMU, a multi-modal benchmark comprising 3.7k scientific diagrams and 18.3k validated questions to evaluate MLLMs on diagram-to-code parsing, editing, and question answering, revealing that while models excel at reasoning, they struggle with code generation tasks unless operating in agentic settings.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a scientist writing a paper about a new discovery. You have a beautiful graph showing your data, a circuit diagram of your experiment, or a molecular structure of a new drug. In the old days, you might have drawn this by hand or used a computer program to make a picture file (like a PNG or JPG) and pasted it into your document. But modern scientists often use a special language called LaTeX to write their papers because it looks so professional and handles math perfectly. Inside LaTeX, there is a super-powerful tool called TikZ. Think of TikZ as a set of very precise instructions for a robot artist. Instead of giving the robot a finished picture, you give it a list of commands like "draw a red circle here," "connect this line to that one," and "write the number 5 here." If the robot follows the instructions perfectly, it draws the exact diagram you need, and it fits perfectly into your paper.
Now, imagine you have a super-smart computer brain called a Multimodal Large Language Model (MLLM). You might think, "If this brain can read a picture and understand what it means, can't it just look at my diagram and write the TikZ instructions for me?" That is the big question this paper tackles. Scientists want these AI brains to be able to look at a diagram, understand the science behind it, and then write the code to recreate it or even change it (like turning a bar chart into a line chart) just by listening to a simple request. This is part of a new trend called "vibe writing," where you just describe what you want, and the AI does the heavy lifting. But does the AI actually have the skills to do this, or does it just pretend to understand?
The researchers behind Diagram-MMU decided to build a giant, rigorous test to find out. They created a benchmark (a standardized test) featuring 3,744 unique scientific diagrams and 18,305 different questions and tasks. These diagrams cover six different scientific worlds: charts (like graphs), flat geometry, 3D shapes, network graphs, chemistry molecules, and electrical circuits. They didn't just ask the AI to look at the picture; they tested it on three specific challenges:
- Parsing: Looking at a diagram and writing the code to draw it from scratch.
- Editing: Looking at a diagram and the code, then changing the code to match a new instruction (like "make the resistor blue" or "turn this parallel circuit into a series one").
- Question Answering: Looking at the diagram and answering a question about the science inside it (like "what is the total resistance?").
They also tested if the AI could act like a helpful assistant that knows when to look up information. They gave the AI a special "search tool" to look up TikZ syntax (the rules of the language) if it got stuck, and they tested if the AI could plan its steps: "First, I need to understand the picture, then I'll look up the rule, then I'll write the code."
So, what did they find? The results were a mix of "wow" and "uh-oh." The AI models are actually really good at thinking about the diagrams. When asked questions like "Which bar is the tallest?" or "Are these two components connected?", the models got it right most of the time (up to 86% accuracy). They understand the science!
However, when it came to writing the code to draw the diagrams, the models struggled significantly. Even the best models only got about 31% to 57% of the basic building blocks (like circles, lines, and text) in the right place. It's like the AI can describe a perfect house in detail, but when it tries to actually build it with bricks, it puts the windows in the roof and the door in the floor. The paper suggests that while these models are great at reasoning, they are still quite bad at the fine-grained visual perception needed to translate a picture into precise code.
The study also looked at whether giving the AI a "search tool" (letting it look up rules) helped. For the editing tasks, it helped a little bit. But for the question-answering tasks, it often made things worse because the AI got confused by too much information or asked the wrong questions. One model, Claude-4.6 Opus, stood out as the most consistent, getting better at all three tasks when given the search tool, but most other models stumbled.
In short, the paper reveals a funny gap: today's AI can be a brilliant scientist who understands your diagrams perfectly, but it's still a clumsy artist when it tries to draw them using the TikZ language. The authors hope this test will help developers build better tools so that in the future, scientists can truly just "vibe" their diagrams into existence without worrying about the code.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.