🤖 machine learning

Symbolic Graphics Programming with Large Language Models

This paper introduces SGP-GenBench to evaluate large language models' ability to generate symbolic graphics programs (SVGs) and proposes a reinforcement learning approach with verifiable rewards that significantly enhances generation quality and semantic alignment, enabling open-source models to match the performance of frontier systems.

Original authors: Yamei Chen, Haoquan Zhang, Yangyi Huang, Zeju Qiu, Kaipeng Zhang, Yandong Wen, Weiyang Liu

Published 2026-08-10

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yamei Chen, Haoquan Zhang, Yangyi Huang, Zeju Qiu, Kaipeng Zhang, Yandong Wen, Weiyang Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to describe a picture to a friend who has never seen it. You might say, "Draw a red car next to a blue bike." But natural language is messy; it's full of wiggle room. Your friend might draw a car that's too big, a bike that's floating in the air, or a red that looks more like pink. This is the daily struggle for computers trying to turn words into images. For a long time, scientists have taught computers to "dream" up pictures using pixels, like a digital painter smudging colors on a canvas. But there's another way: Symbolic Graphics Programming. Instead of smudging pixels, the computer writes a precise recipe—a set of mathematical instructions—to build the image piece by piece, like assembling a LEGO set or following a strict architectural blueprint. This method is incredibly precise, but it's also very hard to get right.

Enter Large Language Models (LLMs). These are the super-smart AI brains that can write code, tell jokes, and answer questions. They are great at following instructions, but can they write the specific "blueprints" needed to build a perfect image from a simple description? That's the big question this paper tackles. The researchers wanted to know: Can these AI brains learn to be master architects of visual scenes, writing the exact code needed to draw a complex world, and if they can't do it well on their own, can we teach them to get better?

The Paper's Story: Teaching AI to Draw with Code

The researchers started by testing the current state of the art. They built a giant test called SGP-GenBench, which is like a massive driving test for AI artists. They gave various AI models a list of prompts, ranging from simple ("a red car") to complex ("three people riding on the backs of elephants"). The goal was to see if the AI could write the code (specifically a type called SVG, or Scalable Vector Graphics) that would render exactly what was asked.

The results were a bit of a mixed bag. The "frontier" models—the super-expensive, closed-source ones used by big tech companies—did pretty well. They could mostly get the shapes and colors right. But the open-source models, which are free for anyone to use and study, struggled. They often wrote code that didn't work at all, or they drew things that looked nothing like the prompt. It was like asking a novice to build a house; they might get the roof on, but the walls would be crooked, or the door might be missing entirely.

So, the team asked: How do we fix the open-source models?

They didn't just feed the models more examples to memorize. Instead, they used a technique called Reinforcement Learning (RL). Think of this like training a dog, but for a computer.

The Attempt: The AI tries to write the code for a picture.
The Check: The computer renders the code into an actual image.
The Reward: A super-smart "judge" (another AI trained to understand pictures) compares the result to the original description.
- If the AI wrote code that crashed or didn't render, it gets a zero.
- If the image looks like the prompt, it gets a high score.
- If the image is close but has a few errors, it gets a medium score.

The AI then uses these scores to learn. It realizes, "Oh, when I put the sun in the top left, the score goes up. When I forget the wheels on the car, the score goes down." Over thousands of tries, the AI learns the rules of the game.

The Surprising Results

The results were impressive. After this "training camp," the open-source model (Qwen-2.5-7B) didn't just get a little better; it jumped to a level where it could compete with the best closed-source models. It went from drawing messy, broken scribbles to creating clean, detailed, and accurate vector graphics.

But the most fascinating part wasn't just the final score; it was how the AI changed its behavior while learning. The researchers watched the AI's "thought process" evolve and found two cool tricks it started using:

Breaking Things Down: At the beginning, the AI tried to draw a complex object (like a motorcycle) in one big, messy block of code. As it learned, it started breaking the motorcycle into smaller, manageable parts: "First, I'll draw the body. Then, I'll add the wheels. Now, let's add the handlebars." It was like a chef who stopped trying to throw all the ingredients in the pot at once and started prepping each one carefully.
Adding "Optional" Details: The AI started adding things that weren't explicitly asked for but made the picture feel more real. If you asked for "people sitting at a table with cake," the AI might start adding sprinkles on the cake or crumbs on the table. If you asked for a beach scene, it might add waves or sand. These weren't mistakes; they were creative choices that made the scene feel complete and natural, showing the AI was starting to understand the context of the scene, not just the literal words.

Why This Matters

The paper suggests that this method is a powerful way to teach computers how to "see" and "draw" without needing millions of perfect examples to start with. By using a reward system that checks if the picture matches the words, the AI learns to align its language skills with its visual skills.

The researchers found that this approach works so well that the open-source model, which started out struggling, ended up performing on par with the most advanced commercial systems. They also discovered that the AI didn't just memorize answers; it learned a strategy of breaking complex tasks into smaller, controllable steps and adding helpful details to make the final result better.

In short, this paper shows that with the right kind of training—where the AI gets immediate feedback on whether its "drawing" matches the description—we can turn a basic language model into a precise, creative, and highly capable graphic programmer. It's a step toward AI that doesn't just guess what a picture should look like, but actually knows how to build it, one precise line of code at a time.

Technical Summary: Symbolic Graphics Programming with Large Language Models

Problem Statement

Large Language Models (LLMs) have demonstrated strong capabilities in program synthesis, yet their ability to generate Symbolic Graphics Programs (SGPs)—specifically code that renders into precise, executable visual content—remains underexplored. Unlike standard text-to-image generation which relies on latent pixel representations, SGPs require the model to translate natural language into formal, structured code (e.g., Scalable Vector Graphics or SVG). This task demands precise spatial reasoning, geometric accuracy, and compositional logic, bridging the gap between abstract linguistic descriptions and deterministic visual outputs. The paper addresses two primary questions:

To what extent can current LLMs effectively generate SGPs?
How can the generation capabilities of LLMs, particularly open-source models, be improved to match frontier proprietary systems?

Methodology

1. Benchmark Construction: SGP-GenBench

To systematically evaluate SGP generation, the authors introduce SGP-GenBench, a large-scale benchmark comprising three complementary components:

Scene Generation (COCO-val): 1,024 examples from MS-COCO evaluating complex scenes with multiple objects and interactions.
Object Generation (SGP-Object): 930 examples focusing on high-fidelity rendering of single objects.
Compositional Generation (SGP-CompBench): 3,200 prompts testing three specific compositional dimensions:
- Attribute Binding: Color, shape, and texture consistency.
- Spatial Relations: 2D, 3D, and implicit spatial relationships.
- Numeracy: Accurate generation of specific object counts (3–10 items).

Evaluation metrics include semantic fidelity (CLIP-Score, DINO-Score, VQA-Score, HPS v2) and compositional quality (judged via specific prompts for attributes, relations, and counting accuracy).

2. Reinforcement Learning with Verifiable Rewards (RLVR)

To improve SGP generation, the authors propose a Reinforcement Learning (RL) approach using Group Relative Policy Optimization (GRPO). The method does not require ground-truth SVG programs or paired image-code datasets. Instead, it leverages external vision foundation models to provide verifiable reward signals.

Reward Design:
The reward function $r(s, c, x)$ for a generated SVG $s$ , caption $c$ , and optional reference image $x$ is factorized into:

Format-Validity Gate ( $r_{fmt}$ ): A binary check ensuring the output follows a specific <THINK>/<ANSWER> structure and successfully renders into a raster image without errors. Invalid outputs receive zero reward.
Perceptual Alignment:
- Text-Image Alignment ( $r_{Text}$ ): Computes the cosine similarity between the caption and the rendered image using a contrastive vision-language encoder (e.g., SigLIP).
- Image-Image Alignment ( $r_{Image}$ ): When a reference image is available, it computes the similarity between the rendered image and the reference using a self-supervised vision encoder (e.g., DINO).

The total reward is a weighted sum of these components, gated by validity. This allows the model to learn from raw images and text prompts without explicit program annotations.

Key Contributions

SGP-GenBench: The introduction of a comprehensive benchmark that evaluates LLMs across object fidelity, scene coherence, and compositional consistency (attributes, spatial relations, and numeracy).
RL-Driven SGP Generation: A novel post-training framework that uses cross-modal alignment rewards (text-image and image-image similarity) to fine-tune LLMs for SVG generation. This approach enables open-source models to achieve performance comparable to frontier proprietary models.
Emergent Behavioral Analysis: An in-depth analysis revealing that RL training induces specific emergent strategies:
- Decomposition: The model learns to break complex objects into simpler, controllable primitives (e.g., separating a motorcycle into wheels, frame, and lights) rather than generating monolithic code blocks.
- Contextual Detailing: The model spontaneously generates semantically relevant, unrequested details (e.g., adding waves to a beach scene or sprinkles to a cake) to enhance scene coherence and realism.

Experimental Results

Benchmark Performance

Proprietary vs. Open-Source: Frontier proprietary models (e.g., Claude 3.7 Sonnet Thinking, Gemini 2.5 Pro) significantly outperform open-source models on SGP-GenBench. Performance correlates strongly with general coding capabilities.
Impact of RL: The RL-finetuned Qwen-2.5-7B model substantially narrows the gap with proprietary systems.
- It improved its overall compositional score from 8.8 (base) to 60.8, outperforming other open-source models like DeepSeek-R1 and QwQ-32B.
- It achieved the highest VQA-Score (0.596) across all models, slightly surpassing Claude 3.7 Sonnet Thinking.
- It demonstrated superior detail fidelity in qualitative comparisons, generating more accurate geometric shapes and scene elements than baseline models.

Ablation Studies

Embedding Models: Using SigLIP as the text-image encoder yielded stronger grounding and diversity compared to CLIP, particularly on real-world image splits (COCO).
Chain-of-Thought (CoT): Explicit CoT prompting was found to be non-essential for quantitative performance; the SVG code itself serves as an implicit reasoning trace.
Best-of-N vs. RL: Analysis of Best-of-N sampling showed that while increasing samples improves performance, the gains from RL training are qualitative and cannot be replicated by standard sampling within practical compute budgets (requiring millions of samples to match the gains of 1,000 RL steps).

Significance and Claims

The paper posits that symbolic graphics programming offers a precise and interpretable lens for studying cross-modal grounding. By forcing LLMs to generate executable code rather than latent pixels, the task exposes specific deficiencies in spatial reasoning and compositional logic.

The authors claim that Reinforcement Learning with Cross-Modal Rewards provides a scalable mechanism for injecting visual knowledge into LLMs without the need for costly, curated image-program datasets. The results demonstrate that open-source models can be trained to approach the capabilities of frontier closed-source systems in generating complex, semantically aligned vector graphics. Furthermore, the emergent behaviors observed (decomposition and contextual detailing) suggest that RL not only optimizes for the reward signal but also fosters a deeper, more structured understanding of visual scenes.