Vector-Bench: Can Models Surgically Edit SVG Code?
The paper introduces Vector-Bench, a rigorous benchmark of 40 SVG repair tasks designed to evaluate instruction-based vector editing, revealing that even the strongest models struggle to surgically apply requested changes without altering unrequested elements, achieving only 15.0% full specification success.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are teaching a robot to be a digital artist. You give it a picture made of code—like a digital drawing built from instructions rather than paint—and you say, "Hey, the sun in this picture is too big; shrink it, but don't touch the clouds or the mountains." This is the world of vector graphics. Unlike a standard photo, which is just a grid of colored dots, a vector image is a living program. It's a list of commands telling a computer how to draw lines, shapes, and colors. Because it's code, if the robot changes the wrong line, it doesn't just make the picture look weird; it might break the whole file, making it impossible to open or edit later.
The big question scientists are asking is: Can these smart AI models actually follow instructions to fix a specific part of this code without messing up the rest? It's a bit like asking a chef to replace the salt in a soup without changing the temperature, the pot, or the recipe for the bread on the side. If the AI just makes the soup taste right but burns the bread, it's a failure, even if the soup looks good. This is the challenge of instruction-based editing: doing exactly what you asked, and only what you asked, while keeping the invisible structure of the code perfect.
Enter Vector-Bench, a new and tricky test designed to see if AI models can really be surgical editors. The researchers created 40 specific puzzles where an SVG image is "corrupted"—maybe a train door is floating in the wrong spot, or a jellyfish has lost its tentacles. They gave the AI models a simple, human-language instruction like, "The amber signal has gone dark; fix it and leave the station alone." The models had to rewrite the code to fix the signal without accidentally moving the train, deleting the station platform, or breaking the file format.
The results were a bit of a reality check. The researchers tested 34 different AI models, ranging from cheap, small ones to the most powerful "frontier" models available. The best-performing model, Claude Sonnet 5, managed to get the entire job done perfectly—fixing the error, keeping everything else safe, and producing a valid file—only 15.0% of the time. That's right: even the smartest AI failed to follow the full instructions in more than 8 out of 10 tries.
However, the story isn't just about failure; it's about how they failed. The paper found that the models were actually quite good at the "fixing" part. On average, they made visible progress on the requested repairs about 43.7% of the time. They could often make the signal brighter or move the door closer. But they were terrible at the "leave the rest alone" part. They frequently renamed things they shouldn't have, shifted unrelated objects, or broke the code structure so the file became invalid.
The study explicitly rules out the idea that "looking good" is the same as "working correctly." A model might produce an image that looks fixed to a human eye, but if the underlying code has been rewritten or damaged, it's a failure. The researchers proved this by using a strict, computerized judge that checks not just the picture, but the code itself. They found that while models can make some repairs, the ability to surgically edit code without collateral damage is still a huge hurdle.
In short, the paper suggests that while AI is getting better at drawing and fixing pictures, it still struggles to be a precise, careful editor of the code that makes those pictures work. The gap between "making a change" and "making the right change without breaking anything else" remains wide. The authors released all their data, code, and results, showing that even the most expensive and powerful models are currently more like enthusiastic interns who fix the problem but accidentally knock over the bookshelf, rather than the precise surgeons we need for editing digital art.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.