ChartAnno: Evaluating MLLMs for Chart Annotation Generation

Imagine you are the captain of a spaceship, and your computer has just drawn a beautiful, complex map of the stars. The map is perfect, but it's silent. It shows the stars, but it doesn't tell you why they are there, which ones are the most dangerous, or where the treasure is hidden. To make the map useful, you need to add notes, arrows, and labels—this is called chart annotation. It's the art of turning a raw picture into a story that anyone can understand.

In the world of artificial intelligence, there are super-smart robots called Multimodal Large Language Models (MLLMs). Think of them as robots that can read text, look at pictures, and write computer code all at the same time. We've already taught them to understand charts and even draw new ones from scratch. But there's a tricky gap: can these robots look at an existing chart and add the right notes to explain it? It's not just about drawing a line; it's about figuring out what the line means and then writing the code to put a label exactly where it belongs. This is the challenge the paper tackles: teaching AI to be a helpful tour guide for data, rather than just a mapmaker.

Enter CHARTANNO, a new "training ground" created by researchers to test how good these AI robots are at this specific job. The researchers built a massive library of 1,200 real-world charts—ranging from news graphics to scientific papers—and paired each one with a set of instructions. They didn't just give the robots one type of instruction; they tested them with three levels of detail, like a video game with increasing difficulty:

1. The "Vague Goal" Level: The robot is told, "Highlight the most important trend." It has to guess what that means and how to show it.
2. The "Action Plan" Level: The robot is told, "Draw a purple box around the years 1890 to 1900 and put a label in the middle." It knows what to do but has to figure out the exact colors and sizes.
3. The "Blueprint" Level: The robot is given a recipe: "Draw a purple box from x=0 to x=10 with a specific shade of purple and place the text at coordinates (4, 15)."

The researchers then asked 10 different AI models (some made by big tech companies, others open-source) to write the computer code that would add these annotations. They checked if the code actually worked, if the chart looked right, if the meaning was clear, and if the design was pretty.

Here is what they found, and it's a bit of a mixed bag. First, the big-tech models (like the ones from Google and OpenAI) are still the champions, generally doing the best job at understanding the "vague goals" and creating beautiful designs. However, some of the open-source models are catching up fast, with one called Kimi K2.5 performing almost as well as the expensive, closed-source giants.

The study also revealed some surprising truths about how these robots think. When the instructions were super specific (the "Blueprint" level), the robots did a great job. But when the instructions were vague (the "Vague Goal" level), they often stumbled, struggling to guess what the most important part of the chart actually was. It's like giving a chef a recipe with exact measurements versus just saying, "Make something delicious"; the robots are great at following recipes but still learning how to be creative chefs.

Another interesting discovery was about what the robots need to see. You might think that showing the robot the picture of the chart would help it a lot. But the researchers found that giving the robot the code that created the chart was far more important. The picture helped a little bit with making the annotations look pretty, but without the code (which contains the actual data numbers and structure), the robots got lost. In fact, if you only showed them the picture and hid the code, their performance crashed. It turns out that for these AI robots, knowing the math behind the picture is more useful than just looking at the picture itself.

Finally, the researchers noted that the task gets much harder as the charts get more complex. If a chart has a lot of data points or requires a long, complicated code change to add the annotation, even the smartest robots start to make mistakes. They might write code that doesn't run, or they might put a label in the wrong spot.

In short, CHARTANNO shows us that while AI is getting very good at drawing and reading charts, teaching it to explain them with annotations is still a work in progress. The robots are excellent at following strict instructions but still need help figuring out the "big picture" on their own. The researchers hope this new test will help build better AI tools that can one day turn any confusing chart into a clear, helpful story for everyone.