💻 computer science

ArtChart: A Benchmark for Faithful Artistic Chart Generation with Integrated Text Rendering

This paper introduces ArtChart, a novel framework and benchmark that achieves faithful artistic chart generation by integrating mathematically precise geometry, accurate in-image text rendering, and coherent stylization through a specialized plug-and-play module and a multi-expert reinforcement learning strategy.

Original authors: Meijia Huang, Yingjie Yin, Shihao Wang, Chenguang Ma

Published 2026-07-20

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Meijia Huang, Yingjie Yin, Shihao Wang, Chenguang Ma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are an artist hired to turn a boring spreadsheet into a masterpiece. You want the data to tell a story, so you turn the bars of a graph into stacks of colorful books or the slices of a pie into delicious fruit. This is the dream of "artistic chart generation." But here is the catch: if you turn a bar representing "100" into a stack of books, that stack must still look exactly like it represents 100, not 50. If you write the word "Apple" next to a fruit, it must be spelled correctly and placed right next to the apple, not floating in the sky or next to a banana.

For a long time, computers have been great at two separate things: making pretty pictures from words (like turning "a cat" into a photo of a cat) and making perfect charts from numbers (like Excel). But when you ask a computer to do both at once—make a chart that is both mathematically perfect and artistically wild—it usually trips over its own feet. It might draw a bar that is the wrong height, spell a number wrong, or put the label "10" next to the bar that should say "100." This paper, called ArtChart, tackles this messy problem. It asks: Can we build an AI that doesn't just make pretty pictures, but makes faithful artistic charts where the math is right, the text is spelled correctly, and the style is stunning?

The Problem: When AI Gets Creative, It Gets Messy

The authors found that current AI models, which are usually very good at following instructions, struggle when asked to generate these specific types of images. They tried asking standard AI models to draw things like "a realistic photo of a horizontal bar chart showing seaweed, crustaceans, and fish with specific numbers." The results were often a disaster. The AI would:

Distort the Math: A bar meant to be twice as tall as another might end up the same size.
Hallucinate Text: It might write "Seaweed" as "Seaweed" or invent numbers that weren't in the instructions.
Mix Up Labels: It might put the number "15" next to the "Fish" bar when it should be next to the "Crustaceans."
Lose Readability: The chart might look so artistic that you can't tell what the data actually means.

The paper argues that generating these charts isn't just about making a pretty picture; it's a complex puzzle where geometry, text, and style must all work together perfectly. If one piece fails, the whole chart is useless.

The Solution: ArtChart's Three-Stage Training

To fix this, the team built a new system called ArtChart. They didn't just throw more data at the problem; they designed a specific training pipeline with three distinct stages, like a student learning to paint a chart.

Stage 1: The Skeleton Builder (Math Control)
First, they taught the AI to understand the "skeleton" of the chart. They gave the AI a text-free, grayscale version of the chart (just the shapes, no colors or words) and asked it to learn how to keep the geometry perfect. Think of this as an architect drawing the blueprints. The AI learned to ensure that if a bar is supposed to be tall, it stays tall, regardless of whether it looks like a book or a carrot. This stage uses a special tool called a "ControlNet" that acts like a rigid guide, forcing the AI to respect the numbers.

Stage 2: The Text and Style Coach (Reinforcement Learning)
Once the skeleton was solid, the AI still made mistakes with words and style. So, the team used a technique called Reinforcement Learning (RL). Imagine a teacher grading the AI's work. If the AI spells a word wrong or puts a label in the wrong spot, the teacher gives a "bad grade" (a low reward). If the text is perfect and the style matches the request, the AI gets a "good grade." The AI tries again and again, learning from these grades to get better at spelling and placing labels correctly.

Stage 3: The Harmony Expert (Multi-Expert Distillation)
Here is where it gets clever. The team realized that being good at math, good at spelling, and good at style are three different skills. Sometimes, focusing too hard on spelling makes the chart look boring, or focusing on style makes the math look weird. To solve this, they trained three separate "expert" AIs: one focused only on math, one on text, and one on style. Then, they combined them into one "student" AI using a method called "distillation." This student learned how to balance all three skills at once, ensuring the final chart was mathematically correct, spelled perfectly, and visually beautiful.

The Results: A New Benchmark

To prove their system worked, the team created a giant test called ArtChart-Bench. This isn't just a few random pictures; it's a massive collection of 2,000 prompts (1,000 in English and 1,000 in Chinese) covering four types of charts (bar, horizontal bar, pie, and area) and 15 different artistic styles. They even included tricky cases, like very long labels or tiny numbers, to see if the AI would break.

They also built ArtChart-Eval, a six-point scoring system to grade the charts on:

Math Logic: Are the shapes the right size?
Text Accuracy: Is the spelling correct?
Text Layout: Are the words in the right place?
Aesthetics: Does it look good?
Instruction Following: Did it do what was asked?
Readability: Can a human actually read the data?

When they tested ArtChart against other top AI models (including some very famous ones), ArtChart won. It scored much higher on math and text accuracy while still looking artistic. For example, while other models struggled to get the numbers right or spelled words incorrectly, ArtChart consistently produced charts where the data was faithful and the text was perfect.

What This Means

The paper suggests that we are getting closer to AI that can handle complex, multi-step creative tasks without losing the details. By separating the "math" part from the "art" part and then carefully blending them, the team showed that it is possible to generate charts that are both beautiful and trustworthy.

However, the authors are careful to note that this isn't a magic wand for every chart yet. Their system currently works best with simple, one-dimensional data (like a list of items and numbers) and four specific chart types. It still relies on that initial grayscale "skeleton" to work perfectly, meaning it doesn't yet prove that AI can figure out the math from scratch just by reading a sentence. But for now, ArtChart sets a new standard, showing that with the right training, AI can be a faithful artist, not just a dreamer.

Technical Summary: ArtChart

Problem Definition

The paper addresses the challenge of faithful artistic chart generation, a task that requires simultaneously satisfying four coupled constraints: preserving numerical geometry, rendering exact in-image text, binding labels to the correct visual marks, and maintaining a coherent artistic style.

While modern text-to-image (T2I) and image-editing models excel at style synthesis and visual text rendering, they frequently fail when applied to chart generation. Pilot experiments identified four recurring failure modes:

Mathematical-logic distortion: Bars with incorrect heights, pie sectors with wrong angles, or incorrect mark counts.
Text-content errors: Hallucinated digits, missing units, or garbled characters.
Text-layout mismatch: Labels placed above the wrong categories or values drifting from their corresponding marks.
Visual-style/readability failure: Over-stylization that renders marks unreadable or a mismatch between the prompt's theme and the output.

Existing solutions, such as programmatic visualization tools (e.g., Matplotlib), lack artistic flexibility, while pictorial chart generation methods often fail to systematically render chart text or treat label-value binding as a secondary annotation rather than a core generation metric.

Methodology: The ArtChart Framework

ArtChart proposes a three-stage training pipeline designed to jointly optimize structural fidelity, text accuracy, and visual aesthetics using the Qwen-Image-2512 backbone.

1. Chart-Specific Mathematical Control Module

To ensure structural and logical fidelity, the authors train a ControlNet branch conditioned on text-free grayscale chart layouts.

Input: A deterministic grayscale image ( $G$ ) containing only mark geometry (e.g., bar heights, pie angles) without axes, ticks, legends, or text.
Process: The ControlNet is trained on triples $(P, G, I^\star)$ , where $P$ is the prompt and $I^\star$ is the target image. The base generator is frozen.
Objective: The controller learns to map the geometric structure of $G$ to the final image, ensuring that the generated marks strictly adhere to the input data values without learning to copy text scaffolding.

2. Reinforcement Learning (RL) for Text and Aesthetic Alignment

After the ControlNet stage, dominant errors shift to text corruption and layout mismatches. The authors freeze the ControlNet and base model, attaching a LoRA adapter to the DiT (Diffusion Transformer) for optimization via Flow-GRPO (Group Relative Policy Optimization).

Rewards:
- OCR Reward ( $R_{ocr}$ ): Uses Hungarian matching on normalized edit distance to align detected text with target titles, categories, and values.
- Layout Reward ( $R_{lay}$ ): Uses a Vision-Language Model (VLM) to verify semantic attachment, checking if category and value slots are in the correct positions relative to the marks.
- Aesthetic Reward ( $R_{aes}$ ): Uses PickScore to evaluate visual quality.
Optimization: The policy is updated using a clipped GRPO objective, converting the deterministic flow ODE into an equivalent SDE to enable policy-gradient optimization.

3. Multi-Expert Distillation (OPD)

To resolve conflicts between competing rewards (e.g., optimizing for text accuracy often degrades aesthetics), the authors employ an On-Policy Distillation (OPD) strategy:

Stage 1: Three single-reward experts ( $\phi_{ocr}, \phi_{pos}, \phi_{aes}$ ) are trained independently to their respective performance ceilings using Flow-GRPO.
Stage 2: A student LoRA is trained via on-policy distillation. A hard router selects the most appropriate teacher expert for each sample, and the student learns from the teacher's velocity field. This approach allows the model to approach the performance ceilings of all three experts simultaneously without the "seesaw effect" of weighted reward sums.

Key Contributions

Task Definition: The paper formally defines artistic chart generation as a faithful image-generation task requiring the integration of numerical geometry, text rendering, label binding, and stylization.
ArtChart Framework: A novel system combining a plug-and-play math control module, RL with complementary rewards, and a multi-expert distillation strategy to balance optimization across conflicting objectives.
ArtChart-Bench: A bilingual (1K English, 1K Chinese) benchmark of 2K prompts spanning four chart types (bar, horizontal-bar, pie, area), 15 artistic styles, and diverse data distributions. It includes diagnostic metadata (e.g., value patterns, label formats) for fine-grained failure analysis.
ArtChart-Eval: A six-axis evaluation suite covering Mathematical Logic, Text Accuracy, Text Layout, Aesthetics, Instruction Following, and Readability. This protocol supports fair comparison across T2I, image-editing, and controllable-generation models under a unified prompt-output standard.

Experimental Results

Extensive experiments on ArtChart-Bench demonstrate that ArtChart consistently outperforms baselines, including pure T2I models (Qwen-Image-2512, FLUX.2), image-editing models, and generic ControlNet approaches.

Overall Performance: ArtChart achieves the highest scores across all six dimensions. Notably, it reaches a Math Logic score of 9.05 (compared to 2.87 for the base T2I model) and a Text Accuracy of 9.54.
Ablation Studies:
- Adding the Chart-Specific ControlNet significantly improves Math Logic (from 2.87 to 9.06) and stabilizes layout.
- GRPO further refines text and layout scores but slightly impacts aesthetics.
- OPD successfully recovers aesthetic quality while retaining the gains in text and layout, demonstrating the efficacy of the multi-expert distillation in resolving reward conflicts.
Chart Type Analysis: The model performs best on bar and horizontal-bar charts. Pie charts remain the most challenging due to sector-angle and label-placement complexities, while area charts show strong math/text scores but slightly lower readability.

Significance and Claims

The paper positions ArtChart as the first work to simultaneously address mathematically faithful chart synthesis, accurate in-image text rendering, and artistic stylization.

Reproducibility and Extensibility: ArtChart-Bench provides a structured testbed with diagnostic metadata, allowing researchers to attribute failures to specific factors (e.g., label length, value magnitude) rather than averaging them out.
Evaluation Standard: ArtChart-Eval establishes a unified protocol for comparing diverse generation paradigms (T2I, editing, control-based) on a task where geometric integrity and text correctness are non-negotiable.
Future Foundation: The authors claim that this work establishes a solid foundation for future research in generating reliable visual communication, moving beyond merely "attractive" images to those that are mathematically and semantically faithful.

The paper concludes by acknowledging limitations, noting that the current approach relies on explicit grayscale conditions rather than inferring geometry purely from text, and that the benchmark is currently limited to one-dimensional categorical data and four chart types.